Novell Cluster Services™ for Linux® and NetWare®

Next Page

Book Cover
Novell Cluster Services™ for Linux® and NetWare®
By Rob Bastiaansen, Sander van Vugt
...............................................
Publisher: Novell Press
Pub Date: May 09, 2006
Print ISBN-10: 0-672-32845-3
Print ISBN-13: 978-0-672-32845-9
Pages: 312

Table of Contents | Index
Overview

Need to configure or manage Novell Cluster Services on NetWare, Linux or a mixed environment? Pick up a copy of the official reference guide, Novell Cluster Services for Linux and NetWare. This book blends in-depth information with practical, real world examples to cover cluster services configuration strategies, backup requirements, cluster services management, and upgrading tactics. You'll gain invaluable insight from authors Rob Bastiaansen and Sander van Vugt, two Novell Certified Instructors with day-to-day experience consulting on the topics covered in this book. Master intalling and managing Novell Cluster Services with the tutorial not available from anyone else, Novell Cluster Services for Linux and NetWare.

Next Page


Table of Contents

Previous Page
Next Page

Book Cover
Novell Cluster Services™ for Linux® and NetWare®
By Rob Bastiaansen, Sander van Vugt
...............................................
Publisher: Novell Press
Pub Date: May 09, 2006
Print ISBN-10: 0-672-32845-3
Print ISBN-13: 978-0-672-32845-9
Pages: 312

Table of Contents | Index

Copyright
About the Authors
Acknowledgments
Introduction
Chapter 1. Introduction to Clustering and High Availability
 | Novell Cluster Services Defined
 | High-Availability Solutions Overview
 | When Not to Cluster Applications
 | Availability Defined
 | High Availability Defined
 | Calculating Average Downtime
 | Avoiding Downtime
 | Novell Cluster Services Requirements
Chapter 2. Examining Novell Cluster Services Architecture
 | Novell Cluster Services Objects and Modules
 | Heartbeats, Epoch Numbers, and the Split Brain Detector
 | Summary
Chapter 3. Clustering Design
 | Cluster Design Guidelines
 | Creating a Failover Matrix
 | Application-Specific Design Guidelines
 | Clustering Without Shared Storage
Chapter 4. Installation and Configuration
 | Installing NCS on OES NetWare
 | Installing NCS on Linux
 | Manually Starting and Stopping NCS
 | Adding a License to Existing Cluster Nodes
 | Removing Nodes from a Cluster
 | Installing NCS in VMware
Chapter 5. Creating Cluster Resources
 | Creating Clustered Volumes
 | Creating Cluster Resources
 | Cluster Script Syntax
 | Summary
Chapter 6. Cluster Management
 | Cluster Resources Management Applications
 | Managing Novell Cluster Services from the Command Line
 | Logging, Event History, and Alerting
 | Summary
Chapter 7. iSCSI
 | iSCSI Introduction
 | Using iSCSI in an Open Enterprise Server Environment
 | iSCSI Target Configuration on OES NetWare
 | iSCSI Initiator Configuration on OES NetWare
 | iSCSI Initiator Configuration on OES Linux
 | Building an Open Enterprise Server Cluster with Shared iSCSI Storage
 | Advanced iSCSI Topics
Chapter 8. Cluster-Enabled Applications
 | Cluster-Enable or Not?
 | Clustering GroupWise
 | Clustering iFolder 3 on OES Linux
 | Clustering the Squid Web Proxy on Linux
 | Clustering the Samba Server on OES Linux
 | Summary
Chapter 9. Advanced Clustering Topics, Maintenance, and Troubleshooting
 | Advanced Configuration and Maintenance
 | Troubleshooting Your Cluster Environment
 | Cluster Services Backup
Chapter 10. Upgrading Clusters
 | Upgrade Methods Explained
 | Technical Upgrade Methods Explained
Chapter 11. Using SUSE Linux Enterprise Server Clustering Options
 | Heartbeat Versus Novell Cluster Services
 | Installing SUSE Linux High-Availability Components
 | Configuring the Shared Storage with a DRBD
 | Clustering the NFS Server with Heartbeat
 | Managing the Shared Resource
 | Avoiding Split Brain
 | Summary
Chapter 12. Introduction to Business Continuity Clustering
 | Business Continuity Clustering Overview
 | Building Your Own BCC Test Environment
 | Summary
Index

Previous Page


Copyright

Novell Cluster Services™ for Linux® and NetWare®

Copyright © 2006 by Novell, Inc.

All rights reserved. No part of this book shall be reproduced, stored in a retrieval system, or transmitted by any means, electronic, mechanical, photocopying, recording, or otherwise, without written permission from the publisher. No patent liability is assumed with respect to the use of the information contained herein. Although every precaution has been taken in the preparation of this book, the publisher and author assume no responsibility for errors or omissions. Nor is any liability assumed for damages resulting from the use of the information contained herein.

Library of Congress Catalog Card Number: 2005928289

Printed in the United States of America

First Printing: May 2006

09 08 07 06 4 3 2 1

ACQUISITIONS EDITOR Jenny Watson
DEVELOPMENT EDITOR Scott Meyers
MANAGING EDITOR Charlotte Clapp
PROJECT EDITOR George E. Nedeff
COPY EDITOR Cheri Clark
INDEXER Cheryl Lenser
PROOFREADER Mike Henry
TECHNICAL EDITOR Josh More
PUBLISHING COORDINATOR Vanessa Evans
BOOK DESIGNER Gary Adair

Trademarks

All terms mentioned in this book that are known to be trademarks or service marks have been appropriately capitalized. Novell Press cannot attest to the accuracy of this information. Use of a term in this book should not be regarded as affecting the validity of any trademark or service mark.

Novell is a registered trademark; Novell Press and the Novell Press logo are trademarks of Novell, Inc. in the United States and other countries. All brand names and product names used in this book are trade names, service marks, trademarks, or registered trademarks of their respective owners.

Warning and Disclaimer

Every effort has been made to make this book as complete and as accurate as possible, but no warranty or fitness is implied. The information provided is on an "as is" basis.

Special and Bulk Sales

Pearson offers excellent discounts on this book when ordered in quantity for bulk purchases or special sales. For more information, please contact

U.S. Corporate and Government Sales
1-800-382-3419
corpsales@pearsontechgroup.com

For sales outside of the U.S., please contact

International Sales
international@pearsoned.com

Novell Press is the exclusive publisher of trade computer technology books that have been authorized by Novell, Inc. Novell Press books are written and reviewed by the world's leading authorities on Novell and related technologies, and are edited, produced, and distributed by the Que/Sams Publishing group of Pearson Education, the worldwide leader in integrated education and computer technology publishing. For more information on Novell Press and Novell Press books, please go to www.novellpress.com.

Associate Publisher Mark Taber
Program Manager, Novell, Inc. Darrin Vandenbos
Marketing Manager Doug Ingersoll


About the Authors

Rob Bastiaansen is an independent consultant and trainer. Rob has a strong focus on NetWare, Cluster Services, eDirectory, and ZENworks. He delivers Advanced Technical Training for Novell in EMEA regarding these topics. Rob is also a technical writer; he writes for several IT magazines in The Netherlands, where he lives. VMware is another of the areas in which Rob works as a consultant and trainer. In 2004, Rob wrote and published his first book, Rob's Guide to Using VMware, which is in print in its second edition. In 2005, Rob published The NetWare Toolbox. He is a Master Certified Novell Instructor and holds all the major Novell certifications, including Certified Linux Engineer and Certified Linux Professional.

Sander van Vugt is an independent trainer and consultant, with a strong focus on Linux in combination with any Novell product available. As a technical trainer, Sander does a lot of work for Novell in the EMEA region where he lives. He specializes in the workings of Linux file systems and file access protocols and clustering services. Sander is also a technical author; he has written 26 books in the Dutch language, and this is his second book in English. Sander is also a Master Certified Novell Instructor and holds all major Novell and Linux certifications.


Acknowledgments

This book could not have been written without the help of many people at Novell. In particular we'd like to thank Robert Wipfel for the time he spent with us thinking about this book. Also we would like to thank Bhogilal Hirani and Brad Rupp for their input related to Chapter 12. Further, we would like to thank Kent Boogert and Richard Jones for their support in writing this book.


Introduction

In this book we provide you with an overview of Novell's high-availability solutions. We cover Novell Cluster Services (NCS) for Open Enterprise Server for both NetWare and Linux. We focus on the latest release of this software, but most information from this book is applicable to older versions of NCS as well. Our goal with this book is to give you a full overview of the process of designing and implementing a cluster. We provide insight into additional technologies such as iSCSI, Linux Heartbeat clustering, and Business Continuity Clustering. We also describe how to install popular applications into your cluster environment. In this book we have tried to go beyond the information provided in the online documentation. We not only describe how to configure NCS clusters, but also pay a good deal of attention to the "why" part of the configuration of a cluster because we would like you to understand what you are doing as well. This book is written for two different audiences. Administrators new to administering NCS, or clustering at all, will find enough information to get themselves going with Novell Cluster Services. And administrators who do already have experience configuring Novell Cluster Services will find that lots of details are included, especially on how to configure NCS on Linux servers.

In Chapter 1, "Introduction to Clustering and High Availability," clustering is introduced. This chapter is aimed in particular at novice cluster administrators, although even experienced administrators may find it helpful. Read this chapter if you want to know more about when you would use a cluster solution and how such a solution should be set up. Chapter 2, "Examining Novell Cluster Services Architecture," goes into the working of Novell Cluster Services; in this chapter we describe how the different components of an NCS network interact to make a working cluster. Also some clustering fundamentals such as Heartbeat packages and the Split Brain Detector (SBD) partitions are covered. This chapter is aimed mainly at those who want to know what's happening behind the scenes. If you are in a hurry and want to start as soon as possible, you can safely skip these two chapters.

Chapter 3, "Clustering Design," is an important one that everyone should read; it is about the design of a cluster. We've put all our field experience and some of the most important field experience of Novell technicians together in this chapter. We consider this one of the most important chapters in the book, and we do recommend that everyone read it, particularly because a proper understanding of how to design a cluster can prevent you from making many errors later.

Chapter 4, "Installation and Configuration," discusses installation of NCS. Read this chapter if you want to see more details on how to install NCS on NetWare, on Linux, or even in a VMware virtual environment. From this chapter, you also will learn how to add and remove nodes from the cluster. Chapter 5, "Creating Clustered Resources," gives an introduction to the configuration of clustered resources. Like Chapter 4, this chapter is particularly useful for people who have never worked with NCS before. Then in Chapter 6, "Cluster Management," you will find an overview of all the ways to administer an NCS environment. It explains how to use Remote Manager and iManager, but also how to use SMTP alerting in an NCS environment. Even experienced administrators of NCS clustering may find some information they weren't aware of yet.

Starting with Chapter 7, "iSCSI," some more advanced topics are covered. Most clusters do need a solution that offers shared storage, and in general iSCSI is considered a good and affordable solution to do just that. From the information in this chapter, you will learn how to configure NetWare as an iSCSI target and connect the iSCSI initiator software on Linux to that iSCSI target. We have chosen not to include any other information on shared storage solutions, the main reason being that almost all SAN solutions are rather proprietary. Chapter 8, "Cluster-Enabled Applications," can be considered the core of this book. In this chapter, which is one of the longest in the book, you can find information on how to configure our selection of applications in an NCS cluster. We discuss configuration of some of our favorites, such as iFolder and GroupWise, but some classics from the Linux environment are also covered in this chapter. Next, Chapter 9, "Advanced Clustering Topics, Maintenance, and Troubleshooting," discusses advanced technologies that can be applied to NCS environments. In this chapter, for example, you can read how to re-create the SBD partition when it fails and how to make a proper backup of your cluster. This chapter typically is meant for an administrator who already knows about the nuts and bolts of Novell Cluster Services and wants to upgrade his tuning and troubleshooting skills.

The last three chapters are optional in many cases. Chapter 10, "Upgrading Clusters," discusses how to upgrade an existing cluster to NCS. Chapter 11, "Using SUSE Linux Enterprise Server Clustering Options," tells you how to use the Heartbeat cluster solution in a SUSE Linux Enterprise Server environment. We can recommend this chapter because in the near future the NCS core will be upgraded to the open-source software from the Heartbeat project.

Finally, in Chapter 12, "Introduction to Business Continuity Clustering," you can read how to create a cluster of clusters using Novell's Business Continuity Services.

We put a lot of effort into writing this book and really hope you enjoy reading it! If you ever have any questions or remarks, you can reach us at the following email addresses:

Rob Bastiaansen: mail@robbastiaansen.nl

Sander van Vugt: mail@sandervanvugt.nl


Chapter 1. Introduction to Clustering and High Availability

In this first chapter we will introduce the general concept of clustering. The Novell Cluster Services (NCS) product will be put into perspective with regard to other high-availability solutions. The terminology around high availability and Novell Cluster Services will also be explained in detail. We will provide guidelines here about when to use cluster products and when not to use them. The hardware and software requirements for Novell Cluster Services will be covered at the end of this chapter.


Novell Cluster Services Defined

This section introduces Novell's clustering software. If you are new to the product, you must certainly read on because if you miss the basics, it will be very hard to understand the chapters on design and installation that follow.

With Novell Cluster Services, 2 to 32 servers can be combined into one system, where every server runs its own tasks and can take over services from another server in case of a failover. The most simple clarification of this is an example of two servers providing access to a data volume on an external storage device, such as a storage area network (SAN). Server one is providing access to the volume on a specific Internet Protocol (IP) address. If that server fails, for example because of a power failure, server two will detect that failure, mount the volume, and pick up the IP address that was used to access the service. Because the addressing remains the same, client computers can continue to work with their data. It depends on the end users' application whether it will have any problems with this failover. But in most cases everything just keeps working.

In this scenario all servers in the cluster are actively running their own tasks; therefore, we call this type of cluster an Active-Active cluster, in comparison with an Active-Passive solution, in which one of the servers is not running any tasks but just waits for another server to fail. An example of the latter solution was Novell's Standby Server. It is also possible to look at the Active-Active terminology from a service perspective.

In the architecture and design chapters you will be able to read more about how this actually works, but we would like to give you a brief overview of Cluster Services components and the terminology of Cluster Services.

Shared Disk Access

The most important component for your cluster is the disk that all your servers will share. That external medium contains the data that your applications work with, and it contains a small area that the servers will use for housekeeping. The shared disk can be anything from a simple shared disk on a Small Computer Systems Interface (SCSI) bus to a SAN or an iSCSI connection to a server that acts as a disk target. Each of these solutions is described in more detail later. We will first take a look at what the shared disk is used for. Figure 1.1 shows a typical cluster of OES (Open Enterprise Server) servers.

Figure 1.1. A typical OES cluster setup.

[View full size image]

First of all, the disk will contain a small partition that we call the Split Brain Detector (SBD) partition. The SBD process is explained later in this section. In short, it is the referee that can remove faulty servers from the cluster to keep the rest healthy. The SBD partition is also used to store administrative information and logging information about the cluster.

Note

It is possible to run a cluster without a shared disk and an SBD partition. In Chapter 3, "Clustering Design," you can read about an example of running Dynamic Host Configuration Protocol (DHCP) on a cluster without shared storage. This can be done because the data that's needed to run a DHCP server is stored in eDirectory, a service that is already available on all servers. But you should keep in mind that without the shared disk there is no process in charge of guarding you against a split brain situation.

The remaining part of the shared disk will be used to store data for your clustered services. Separate parts of the data can be assigned to individual services on a per-pool basis. In older versions of the clustering software, the smallest entity to assign to a service and to failover was a volume. But starting with NetWare 6, all Novell Storage Services (NSS) volumes became logical volumes that reside somewhere inside an NSS Pool. Therefore, you will always perform what we call a pool-based failover. An NSS Pool can be active on only one server at a time. Accessing the data and especially writing to it from two servers at the same time would corrupt the data in the pool.

Secondary IP Addresses

Besides the shared disk there is another configuration component you should know about. If your services were running on the primary IP address of a server and the server failed, the clients would lose access to that IP address and therefore also to the service. The other server would, of course, have its own private primary IP address.

To solve this issue, we use secondary IP addresses on the cluster. Every service that we want to fail over to another node will get its own IP address. The address must be one that fits into the subnet of the primary address of the server. And it must also fit into the subnet of the destination server. From this behavior follows the requirement that cluster nodes must always be inside the same subnet.

Using these additional IP addresses also means that you should shift your focus from working with servers to working with services. You no longer access data on the volume of a server but on the volume of a virtual cluster server, either via its eDirectory object or via its IP address or domain name services (DNS) name.

Clustering Terminology

Throughout the book we will use the terminology also used by Novell in its documentation to discuss the clustering components. All these terms are defined here. The listing will also give you an overview of the building blocks for a cluster.

Node

This is what you normally would call a server in your environment. But as soon as it becomes part of a cluster, we use this term to identify a server. A cluster resource can be active on one node and it will fail over to other nodes when the original node fails.

Master and Slave Nodes

In the entire cluster there will always be one node that is the master node. The other nodes are called slaves. You can identify the master node in the management interface because it has a yellow dot in its icon. The only meaning of the master node is that it is the one that holds the IP address that is general for the cluster. Besides, in a split brain scenario with an even tie, which is described later in this chapter, the side where the master node resides wins. From a technical perspective this node is the one that sends out the cluster heartbeat packets to the other nodes as a broadcast, whereas the slave nodes respond to that heartbeat with a unicast response. The master node is the first node to start the Cluster Services software in a cluster. If the server fails, another node will become the master and will also keep the role of master node until it fails or leaves the cluster.

Heartbeat

All nodes in the cluster will communicate via the network to check whether all nodes are still alive. They do this by means of heartbeat packets. When a node stops responding to the heartbeat packets, there is a timeout to wait for the server to respond. If it does not respond after that period, the server will be sent a poison pill.

Master Ip Address

As soon as the first node comes up in the cluster, it will become the master node, and it will bind this IP address to the network interface. It needs to be an address that fits into the subnet of the server's primary address.

Cluster Container

All objects that belong to the cluster will be placed into this container. Also, the object itself will contain general configuration options for the cluster. Every node in the cluster must have access to this container in a local replica, so it is therefore common practice to split off this container as a partition and store a replica on each node.

Cluster Resource

This is an eDirectory object that defines an IP address, a load script, and an unload script for a service that runs in the cluster. It is possible that it also activates a cluster pool and its volumes to be used by the application. Because most applications require access to data, this is the most commonly used configuration. When an application uses data from volumes on the cluster-enabled pool, that application must also be loaded and unloaded from the scripts in this object and not from their own resource object. In Figure 1.2 you can see a general overview of a typical cluster with two nodes and a shared disk pool.

Figure 1.2. An overview of cluster objects for a basic cluster.

[View full size image]

Load and Unload Scripts

Every cluster resource and cluster volume contains a load script and an unload script. In these scripts you can use commands that you could also use on the server's command line or in NetWare Command File (NCF) files. The scripts will contain some default lines for the clustering software, but for the rest it is up to you to define the commands needed to start and stop the application. The cluster load script syntax is covered in depth in Chapter 5, "Creating Cluster Resources."

Cluster Virtual Server

For every pool that you enable in the cluster, you must specify a unique IP address that can be used to access the data but also a virtual server name. The CVSBIND ADD command in the load script activates this server name in the SLP context to make it available on the network. The server is removed from the unload script with the CVSBIND DEL command.

Cluster Template

The resources described previously can be created via a template. These templates contain an example of how a cluster resource can be configured. The cluster container holds several default templates that can be used to create network services such as a DHCP server or an iFolder server.

Split Brain Detector

We have discussed before that a server works with a shared disk where a small partition is created for the Split Brain Detector process. This process is needed to prevent a scenario in which servers in a cluster can no longer see each other via the network and therefore would decide that they each are the only surviving node after a failure. This scenario would result in two clusters being alive with the same name and configuration trying to access the same data. If servers no longer see each other via the network, they will use the shared disk to determine which servers are still alive and which are not. A server that stops responding to heartbeats will be given a poison pill.

Poison Pill

This is a flag that will be set on the shared disk that will be seen by the node that stopped operating via the network. That node will then eat the poison pill and shut down. This is designed this way to prevent that server from accessing the shared data in an uncontrolled way, preventing data corruption.

Failover

This process forms the heart of the entire clustering solution. If a failure on a node is detected, it is this process that will run the load script to activate the disk on another node and load the application. The process will take into consideration what the preferences are for the failover and will decide on what other node the application will be run.

Migration

The process of a failover can also be executed manually. This can be done, for example, if you want an application to run on another server to better balance the load of your environment. Or, for example, if you want to move all applications to other nodes because you want to upgrade a particular node. The difference with the failover process is that the cluster software will run the unload script and execute the load script on the new server.

Failback

After a failover, the resource can also be moved back to where it came from. In that case the unload script will execute and the load script will run on the destination node. This process can be activated automatically or manually. For the manual mode the difference with a manual migration is that using the failback feature will mark the resource as failed, and it will show that administrator intervention is needed to move the resource back to its original node. You might want to do that after business hours, or at least after investigating why a failover occurred. It is because of that last reason that we generally do not configure automatic failback. If you do not know why the resource failed, it could fail again, and thus it could start going back and forth between two nodes.

Epoch

When you're looking at your Cluster Services management software, this number shows you how many times the status of the cluster has changed. When the cluster starts, the epoch is zero, and every time a node joins or leaves the cluster, the epoch is increased by one.

Lives

This number also shows up in your management application. Every cluster resource or cluster volume starts with one life, and every time the resource is brought online or migrated to another node or does a failover to another node, the lives counter is increased by one.

Membership Quorum and Timeout

Starting the cluster software when the entire cluster has been down will start with one node, followed by the other nodes. It would not be a good idea if that first node would start all the cluster resources on its own; the server would probably become too busy. That is the reason why there is a quorum defined in the cluster container object. When the number of nodes defined there is online, the nodes will start loading cluster resources. There is also a timeout value that defines after what period of time the server will start loading resources anyhow. This setting can be overruled on every single resource if you want that resource to come online even when only one node is started. By default, this is the case for the master IP address resource.


High-Availability Solutions Overview

In the preceding section we introduced you to Novell Cluster Services. While talking about that product, we have not covered the entire high-availability portfolio of Novell. We will cover the available solutions in this chapter and we will also give you a sneak peek into the future of high-availability products.

Before we go into the details of all these products, we would like to explain the difference between the types of clusters that are available. This one term, "cluster," is used for different types of clusters and they are not all the same. The cluster that we talk about in this book is what is generally called a high-availability cluster, or a failover cluster. If one node fails, another node will take over its job. And in a healthy cluster environment all nodes are running their own individual services. The other type of cluster comes in different names: high-performance clusters, load-balancing clusters, computational clusters, datacenter clusters, and so forth. What is different for these clusters compared to the ones we write about in this book is that all servers in the cluster work together to deliver the same service and with that share the load of running the service. The entire collection of machines works together to run the application. If a node fails, there is no such a thing as a failover. The service remains running on the nodes that are still alive, and there is absolutely no interruption for the application. An example of such a cluster is the PolyServe Matrix Server, which we will discuss later in this section.

So if these clusters are so different and there is no failover or interruption to the end user, why do we not run all our applications on such a cluster? The answer comes in two parts. Part one is that the software we generally run on Open Enterprise Server does not run on the operating systems that are used to build these types of clusters. They do not run GroupWise Post Office NetWare Loadable Modules (NLMs), for example. The second part of the answer is that these clusters are more expensivenot only because hardware and software have a different price tag, but also because the solution is more complex and requires a higher level of experience from the administrators.

Having said all that, let's take a look at the clustering solutions available today.

Novell Cluster Services

This solution has already been discussed earlier in this chapter, but we would like to look at a few of the benefits of this solution to put it into perspective with the other products in this section.

Mainstream Hardware

One of the great benefits of NCS compared to other high-availability solutions is that it runs on regular server-class hardware. You can buy the same type of servers that you would normally buy to run your applications. You might only want to buy a few more, depending on your design. Also, for the shared storage the hardware availability is much better than it was when NCS was first shipped in 1999. A SAN really was a complex and expensive piece of equipment, and that has changed throughout the years. Nowadays, iSCSI or Distributed Replicated Block Devices can be used as cheap shared storage solutions.

Edirectory Enabled

Servers that run a clustering solution without this extra layer of management are hard to configure and to keep in sync with each other, configurationwise. NCS servers read their configuration from eDirectory, and because this is already a distributed database with built-in fault tolerance, the configuration information is always available.

iSCSI Support

Open Enterprise Server comes with support for iSCSI and therefore offers you the possibility to build a high-availability solution at a relatively low cost. You can use existing hardware and networking equipment to build an iSCSI target that works as a shared disk device for your cluster nodesif your hardware meets the requirements, that is. The old 10MB hub and Compaq Proliant 1600 server might not be your best choice for this. You can read about the requirements you must meet for your iSCSI implementation in Chapter 7, "iSCSI."

Quickly and Easily Expand Cluster Capacity

With NCS it is simple to add a node to an already running cluster. When there is a need to expand the capacity of the cluster, a server is added very easily. And because the storage is external to the servers, increasing the storage capacity is as simple as what your storage vendor offers. NCS does not limit that in any way.

Running a Cluster with Mixed Operating Systems

With Open Enterprise Server it is possible to run cluster nodes in a mixed environment with both NetWare and Linux. This feature has not been added to run this as a permanent solution, but it will help you migrate from a running NetWare cluster to a Linux cluster. You can add new Linux nodes and migrate your data and resources to the new environment.

Business Continuity Clustering

Business Continuity Clustering is based on Novell Cluster Services, and it simply is a cluster of clusters. With this solution, if a cluster fails, a cluster at another site can take over its services and continue operations. Figure 1.3 provides a graphical overview of a basic Business Continuity Cluster (BCC). We will discuss the architecture in this section to the level that you can understand what BCC is. Chapter 12, "Introduction to Business Continuity Clustering," contains a full technical and functional description of what BCC is and how to set up a solution for testing and development.

Figure 1.3. With BCC a complete cluster can fail over to another site.

The basis of a BCC solution is formed of two regular NCS clusters, with a maximum of four. Each cluster runs its own services and provides active services to its own clients. Of course, you can also build the second or third cluster with only failover in mind and not use it in any way. That depends on your design and your demand for high availability.

With the two clusters set up, the next component that is part of BCC is Novell Identity Manager. With this DirXML-based product the configuration of your cluster resources is continuously synchronized between the two geographical sites. When a failover has to occur, the cluster at the remote site will have the entire configuration available to run the services from the original cluster. All the configuration information that has to be changed will be modified during the synchronization cycle. An example of this is the IP addresses for your services. When you do a failover to another site, the IP addresses will need to be in another subnet, and Identity Manager driver takes care of that.

The third component that you see in Figure 1.3 is the SAN. The data you store in the cluster must also be available on the remote site. Therefore the data must be synchronized in some way. There are three general ways of doing this:

  • SAN-based mirroring

  • Host operating systembased mirroring

  • Snapshot-based solutions

Your choice of a data transport solution is limited to factors like available bandwidth and cost. But there are also technical factors to take into consideration. With the mirroring solutions the loss of data is almost zero. Only the data that is still in cache will be lost. With a snapshot solution the data that was generated since the last snapshot point will be lost.

The fourth and last component in Figure 1.3 is the clients. For those there are two things you should look into. The first one is technical. What will have to be done with the IP addressing, for example? If your clients are still on the original subnet, how do they access the services that have failed over to the remote site? But the most important question is this: Where are your clients? Are they still there? If you have to do a failover because you had to evacuate the area, where do you continue your business with your clients? Preparing for this is just as important as setting up the technical stuff.

PolyServe Matrix Server

This high-availability product belongs to the category of datacenter clusters that we described at the beginning of this section. It is a clustering solution for SUSE Linux Enterprise Server that can run applications like Oracle databases or web servers. The applications can run on several nodes in the cluster simultaneously. Figure 1.4 shows a typical PolyServe cluster with NFS exports. With that you can balance the load of an application, and when a node fails the application remains available.

With PolyServe Matrix Server the load of applications can be balanced on servers. With this solution consolidated servers can run more applications than you can normally do on individual servers. The overall performance of any number of servers will improve significantly when they are combined into one cluster. Another benefit of this consolidation is that storage is also consolidated. The benefit of that is also increased storage efficiency.

The product can be bought from Novell. Besides the basic offering, there are also two solution packs that add functionality for specific deployments to your environment. These are the Database Solution Pack and the File Serving Solution Pack.

Figure 1.4. With PolyServe Matrix Server, applications can run on multiple nodes simultaneously.

[View full size image]

Heartbeat Subsystem for High-Availability Linux

The Heartbeat package comes with SUSE Linux Enterprise Server by default. It provides a high-availability solution for Linux applications that run on SUSE Linux. The product originates from the Linux High Availability project. The software provides very similar functionality as NCS. It provides failover clustering for applications. This product is discussed in more detail in Chapter 10, "Upgrading Clusters."


When Not to Cluster Applications

What is this? A section in a clustering book that tells you when not to cluster applications? Do you need that? Yes, you do. Because after reading the previous sections about what clustering is and what the benefits are, you might be tempted to start clustering just for the fun of it.

We are serious here. We have seen it many times. Starting with NetWare 6, a two-node cluster is included with the basic license. That means that it is relatively inexpensive to build a high-availability solution. Several vendors have sold, and some still do sell, servers with a shared SCSI solution for two servers. The cost is not much higher than for two individual servers.

There is nothing wrong per se with using such a cluster, but you should not do it to replace two servers in your environment. In Chapter 3, we will go into more detail about this topic, but for clusters up to six you should always calculate one extra server compared to what you would use in real life.

In a scenario in which you already have two servers that run several services and you build a two-node cluster that runs all those services, what happens if one of the nodes fails? One of the two servers will then have to run the full load of all applications. That might be too much and it could cause your applications to fail, for example, because of a lack of memory.

Planning for a clustering solution is really something you should do with high availability in mind for the applications that are important for your users or for which you have agreed on high availability in a Service Level Agreement. If you do not focus on this and just cluster-enable everything you have running today, you may find yourself with an overloaded server one day, losing valuable uptime for your system.

In many cases it does not even make sense to cluster-enable an application. ZENworks for Desktops is one of the applications that we get a lot of questions about and that was not cluster-enabled in earlier versions. But it actually was. The two main components for ZENworks for Desktops are application management and policy management. For both there are no services that run on the server. All information comes from eDirectory and the file system. And it is absolutely possible to point to a cluster virtual server to deploy software packages and policies. The problem lies in the back-end services. The question is this: Do they need to run on a cluster? We cannot answer that question for you. You should think about the need to cluster-enable them yourself. How important is it that your inventory database has a continuous availability? Do you want to run the Automatic Workstation Import service with high availability? If you have a separate server for these services, they might as well have good availability anyhow, and they are not mission critical. An already imported workstation works fine if it cannot contact the import server at boot time. The only thing that would not happen is the update of workstation information. So you could miss the change in IP address in case you want to remote-control the machine.

This was really just an example of how to look at your services: with a clear mind toward high availability. It is okay with us if you cluster-enable all services in your environment up to the MONITOR.NLM as long as you make sure that you dimension your cluster servers with all these services in mind.


Availability Defined

System administrators and end users do not look at high availability from the same perspective. Where the system administrator sees a server that is available, it does not mean that the end user has available service. This is the most common misconception when we talk to system administrators and IT managers when they are looking into introducing high-availability solutions into their environment. Installing a cluster does not provide instant high availability. There is more to the definition that is important to look at.

Let us first define availability. If you look it up in the dictionary, you will find that available means present or ready for immediate use. And this is really the heart of everything we will be talking about in this section: services being available. We will use the end user's perspective here because that is all that matters. The users and the services they need are the sole reason the IT infrastructure exists. There is no company or organization that runs servers just for the fun of running servers. For the end user availability means this: I can use the service that I am allowed to have access to. This can be something as simple as a word processing application or a spreadsheet program. Or it can be a more complex service such as email and document management, or accounting software. So determining whether a service is available is really simple from the perspective of the end user: Can I access my email? Answering this question with yes means availability, answering with no means nonavailability.

To the end users, it does not matter if a server was down, if a network switch was broken, or if the post office was closed. They cannot access their email and therefore there is no availability.

If you look at your IT infrastructure from this perspective, you should be able to see that there are more components you need to look into than just clustering your servers. Of course, that is the topic of this book. Later in this chapter we will discuss what else you need to look into to improve the level of availability of your IT infrastructure.


High Availability Defined

Now that we have looked into what availability is, we can explore the world of high availability. The use of the word high implies that we get more than normal availability, but clearly not availability without any disruption. Otherwise, we would have used a term such as nonstop availability or continuous availability. No, instead we call it high availability. How high depends on the solution you implement.

In general, vendors of high-availability solutions will talk about the level they offer in terms of a percentage of downtime. It is really impossible to just look at the numbers and decide which solution is the best. An example is a vendor that claims 100% availability. The offering includes redundant hardware with up to three main boards, multiple network interface cards, and so on. You might ask yourself what happens when I need to reboot the server? Does that not lower the percentage of availability of the system? According to this vendor it does not because he means that your system will be available during the business hours that are available and you will have to keep a maintenance window open for such operations.

So if you are looking into the percentage of availability you need, you will first have to do the math yourself and gather all the details.

Are there solutions available that will give me that 100% availability 365 days per year, 24 hours per day? Yes, there are. But they are really a completely different solution than the Novell clustering offering we will be looking into in this book. Examples of servers that have an extremely high level of availability are the ones being used by financial institutions that run payment services for ATMs and payment consoles in shops and restaurants. A downtime of 30 minutes on a busy Saturday afternoon will be guaranteed to make it into the evening news. So the machines that are being used are very reliable, and mostly they are also clustered in a way that will never let the services stop.

What you could read in the preceding section is that there are several types of clusters. The solutions described in the preceding paragraph are not based on failover clustering but on clusters of servers that keep one or more tasks running, in which no single machine has a critical role. That type of clustering allows for the highest availability possible. Of course, the price tag for such a solution is also the highest possible. If the IT manager or the users ever complain about a maintenance window from midnight to 6 a.m. or about a downtime during business hours of an hour per year, you can do the math for them to determine what that last hour of availability would cost compared to the overall cost when you choose to have a little lower availability.

Let us now do the math to see what level of availability we get if we choose a solution with a specific number of nines behind the decimal point. In Table 1.1 we have calculated the maximum downtime a given number of nines represents.

Table 1.1. Availability for Several Grades of Nines for a Full Year

Percentage of Availability

Maximum Downtime

90

36.5 days

99

3.65 days

99.9

8 hours and 45 minutes

99.99

52.5 minutes

99.999

5.25 minutes

99.9999

31.5 seconds

The table tells us what availability a specific number of nines gives us in one year. But is this realistic? If we buy a solution for Novell Cluster Services that promises 99.999% uptime per year, is that realistic? If we need to upgrade the Cluster Services software, can we do that in 5.25 minutes?

If you need realistic numbers, you should not calculate the availability based on one full year. You should look at your business hours. Do your users need 24-hour access, 7 days a week? Or do you have a maintenance window each night that you can use to update software or to test new configurations? Table 1.2 gives a few examples of an uptime window of only 18 hours per day. This makes a great difference compared to a 24-hour window.

Table 1.2. Availability for Several Grades of Nines for 18-Hour Days Including the Weekends

Percentage of Availability

Maximum Downtime

90

27.4 days

99

2.74 days

99.9

6 hours and 34 minutes

99.99

39.4 minutes

99.999

3.9 minutes

99.9999

23.7 seconds

We have not put these figures into the book to discuss the amount of downtime you will get with a Novell cluster solution. The reason that these numbers are important is the Service Level Agreement that you have with your customers (your users in most cases). You need to be able to set up an agreement in which you define how much downtime is allowed for your environment.

The downtime you might get for a system is important, but if you, for example, set up an SLA with your customers with a maximum downtime based on 99.9%, based on an 18-hour working day, including the weekends, you must be able to calculate whether that is realistic. This percentage will give you a time slot to work with to repair the system of a maximum 6 hours and 34 minutes. Anyone who ever had to wait for a new hard disk to arrive knows that you cannot have this happen more than once or twice per year. Of course, we will look into more high availability than just clusters and hard disks, but we will do that later. First we will look into how to calculate what you can expect in terms of downtime when a component fails.


Calculating Average Downtime

There are two terms you should know when you need to calculate what your downtime can be if a component in your environment fails. They are Mean Time Between Failure (MTBF) and Mean Time to Repair (MTTR).

The MTBF is the average time it takes for a component to fail. Let us, for example, take a look at a randomly chosen component: the Intel SRCU42L RAID Controller. For this controller Intel reports a MTBF of 807,267 hours. If we do the math, what we will learn is that this adapter will fail, on average, after 92 years. Not really something to worry about, one might say. But of course this is on average. There remains a chance that you might be one of the unlucky ones on the low end of the scale of the average calculation. And, of course, these numbers are valid only when the equipment is being used in optimal environments. Intel has based its numbers on equipment that runs under a working temperature of 55 degrees Centigrade, or 131 degrees Fahrenheit.

The other statistical number you will need in order to calculate your average availability is the MTTR. It will tell you how much time it will take to fix the problem, by either repairing or replacing the component that fails. This is not something you can get from the vendor's website. It will depend on your hardware and software support contract. Will your supplier replace a component in two hours? Four hours? And what will you do in case of a software problem? Those are not always as easily fixed as hardware problems.

So now that we have both numbers, how do we calculate the average downtime? The formula is simple:

Availability = MTBF / (MTBF + MTTR)

The most important factor here is MTTR. That is also why it is almost always safe to say that the MTTR equals your average downtime if the MTTR is on the low side. The higher the MTTR, the lower the availability. In our example of the Intel RAID Controller, the formula gives an unbelievable 99,9995% of availability. But of course, this is theoretically, based on the entire lifetime of the device, which was 92 years.

It is therefore much more important to look at practical figures and create your own reference of failures and the time it took you to fix the problem. This is really something where ITIL (IT Infrastructure Library) can help out. If you implement it correctly, you will be able to use the Incident Management and Problem Management processes to build a history of your IT infrastructure, and with that data you will be able to determine what your average availability is. Armed with that information, you can start the battle of setting up a Service Level Agreement with your users.


Avoiding Downtime

After all this talk about downtime, it is about time to look into uptime. How can we improve our environment so that it is available to our users for a maximum amount of time? Because this book is about Novell Cluster Services, that is of course one of the components that we champion to use in your environment. But there is much more to it.

Note

It is sometimes really simple to improve availability. We once worked at a customer site where the server room was in one building and the users were divided over two buildings. In the building where the server room was located, a power failure occurred. Of course, all clients stopped working there, except the laptops. And, of course, the clients in the other building also just kept working. The servers were attached to an Uninterruptible Power Supply (UPS) and were configured to go down after 15 minutes. That would have given all the users at least another 15 minutes to continue working and save their work, in case the power was not restored within those 15 minutes. But things went a little differently because the networking equipment was not on a UPS. Had it been then, the availability for laptop users and those in the second building would have been a lot higher.

There are many components in your environment that can fail and that influence the availability of your systemsincluding you because you are only human and nobody is perfect. Even Superman did not see it coming when Lex Luthor found Kryptonite and came after him. That had quite a long Mean Time to Repair, by the way.

In this section we list a number of components for your environment that we think are important to look into if you want to improve your overall availability.

Hardware

This category contains the most obvious causes of failures, and also the most trustworthy. Every end user and IT manager will understand what it means when you tell them that a hard disk has failed. Not so for an abend. There are several actions you can take if you want your environment to be more reliable, which will increase your availability:

  • Buy reliable server-class hardware

  • Use redundant components such as power supplies and fans

  • Use network interface cards (NICs) that support NIC Teaming

  • Apply disk redundancy by means of OS mirroring or RAID

  • Implement redundancy into your SAN and networking equipment

You can also increase availability by using virtualization techniques such as Xen and VMware ESX Server. These solutions remove the hardware layer from your operating system and provide a reliable hardware platform where you run your servers as virtual machines. This way you do not have to implement hardware redundancy in every server, but you just implement that on the bare virtualization hardware.

Environment

In this category you find all the loose ends that can help you improve availability. Serious administrators may laugh about some of them, but believe us when we say that we have seen it all, and worse. So implement these solutions in your environment:

  • Attach critical equipment to a reliable power source and use a backup such as a UPS. Also, do not attach noncritical equipment to a UPS, to prevent using unnecessary resources from your power supply.

  • Work neatly and do not just plug cables into patch panels and let them cross the room as trip wires.

  • Secure access to the server room to prevent unauthorized personnel from accidentally breaking something. Everyone knows some anecdote in which a cleaning lady unplugged the server to plug in the vacuum cleaner.

  • Secure the server room to prevent theft and burglary. It is very shaming if you have to tell that the server is down because the disks have been stolen.

  • Implement a cooling solution that keeps all your equipment operating within the specified temperature bandwidth. And do not forget to monitor the temperature inside the servers with the server vendors' hardware monitoring tools.

  • When planning your server room, look at the big picture. Is your area below sea level? Do not put servers in the basement. The second or third floor is probably always your best choice.

Software

This is probably the category with the hardest problems. For software failures it is sometimes very hard to track down the cause. And whereas you can easily replace a hard disk or a network adapter, it is not always possible to replace a component in a software solution with multiple libraries and executables. There are, however, things you can do to prevent software failures:

  • Test, test, test before you implement new software or a new version into your production environment.

  • Keep your software, including the operating system and drivers, at a reasonably current level.

  • Follow your procedures for changes (see the next section).

  • Implement a monitoring solution to watch what is going on and document what actions to take when error conditions or warnings are reported.

  • Be educated. Make sure that you know the details of your software solutions so that you can fix problems if they occur and that you know where to look.

Procedures

You can set up your environment with the best hardware and software available, but if you do not work with it in a professional way, you can still cause problems yourself. With these simple guidelines you can improve the overall quality and, with that, the availability of your environment:

  • Set up procedures to work with servers and services, and make sure that all administrators follow these procedures.

  • Implement procedures to implement changes. The ITIL framework offers Change Management for this, which could be a good start.

  • Document your environment so that all administrators know where to look for problems when they occur.

  • Document your service agreements for hardware and software so that all administrators know what they offer in case of a failure.


Novell Cluster Services Requirements

When you are going to build a cluster you will need two Open Enterprise Servers, either Linux or NetWare. But you will need to start with a NetWare-only cluster or a Linux-only cluster. You can work toward a mixed cluster environment only when you have a running nonmixed cluster. It also does not make sense to build a mixed clustered environment because this type of clustering is really intended to migrate from one platform to another and not as a permanent solution.

Hardware Requirements

The hardware requirements for your individual nodes are the same as for Open Enterprise Servers that run as individual servers. You must follow the normal guidelines for scaling your server for the applications that you will be running on the cluster. A few tips can be found in the design chapter. In terms of memory and disk space, you really do not have to calculate additional resources for the clustering software. The software takes up only a few megabytes of RAM, and the software is already installed by default on your NetWare server; if you choose to install it on your Linux server, it takes up only a few megabytes.

Important hardware requirements exist for the shared disk system. There are several types of shared disks that can be used. But whatever system you choose, you must meet the following requirements before you start Novell Cluster Services installation:

  • The servers must both have access to the shared disk.

  • A minimum of 15MB must be available on the shared disk. (The SBD partition will actually use one cylinder of the disk so you can not determine the exact size.)

  • Every server needs its own system disk. This can be an internal disk or a LUN (Logical Unit) in the SAN to boot from, as long as it is not shared.

When these requirements have been met, you can start the installation.

Tip

If you want to make sure that the hardware you are using is tested and supported by Novell, you can check this through Novell's Yes Tested and Approved program. In this program for Novell Partners, hardware is tested and when found okay receives the approved status. Information about this program can be found at http://www.novell.com/partners/yes. On this page you will also find a link to search the YES Program bulletins. You can search on product number or description of your hardware.

For this book we assume that your hardware is in place when you start your installation. With the number of hardware vendors that sell SAN solutions, we really cannot go into details for each SAN. The only shared disk solution we will discuss is the one where you implement iSCSI because this solution comes from Novell.

Tip

One other shared disk solution we will discuss is for VMware. If you need to create a cluster for educational purposes or testing purposes, that is a great solution. But also if you have a high-end product like VMware ESX Server, it is also possible to create a cluster with virtual machines and shared storage.

Software Requirements

The software for building a cluster comes with Open Enterprise Server. There are a few requirements you must meet when you start building your cluster:

  • You will need an IP address to assign to the master resource.

  • All servers will need to be in the same subnet.

  • All servers must run the same version of OES and service pack.

  • You need IP addresses for the services you will be running.

  • All servers in the cluster must be in the same tree.

  • For NetWare nodes the servers must all use NSS as the file system. For Linux nodes you can also use Linux file systems, but NSS must be installed on your system.

  • A Windows workstation is needed for a NetWare cluster installation.


Chapter 2. Examining Novell Cluster Services Architecture

In the first chapter of this book, Novell Cluster Services (NCS) was introduced from a high-level overview perspective. Before the next chapters of this book talk about the design of a clustered network and the installation and configuration of Novell Cluster Services, this chapter will discuss what's happening at the background of your Novell cluster. In this chapter you will learn how the different components of an NCS network work together to provide high availability. You will learn about the modules that are used in a cluster, as well as the working of some important background processes. In the second part of this chapter, you will learn how heartbeats, epoch numbers, and the Split Brain Detector (SPD) work together to provide high availability.


Novell Cluster Services Objects and Modules

Novell Cluster Services is present in two different parts of your network. First, you will find cluster-related eDirectory objects. These objects communicate with binaries and libraries that are present on the server. Sometimes they read configuration files to retrieve information, and other times this information is present in the eDirectory object itself as an attribute. In this section we will present an overview of all the important modules and their related eDirectory objects.

Cluster eDirectory Objects

At the highest level, each cluster has a cluster container object in eDirectory. Within the cluster container object, cluster nodes, resources, and templates are created.

Cluster Container

This object is used for storage of all important configuration parameters of the cluster. All of these parameters are clusterwide settings that need to be set only once. The most important of these parameters are included in the following list. Consult Chapter 6, "Cluster Management," for more details about these parameters:

  • Quorum Trigger: This parameter specifies the number of nodes that must be active in the cluster before any cluster resources can be started. This number is also referred to as the quorum membership.

  • Heartbeat Settings: In a clustered environment, nodes need to communicate with each other. Without this communication, it would be impossible to find out whether a node has left the cluster. This communication takes place by means of the heartbeat protocol: By default, all servers send a heartbeat packet once a second. If within a period of eight seconds a node has not been heard from, it will be cast out of the cluster.

  • Cluster Management Port: This is an automatically assigned port address, used for management of the cluster.

Cluster Node

Every physical servers that is a member of the cluster is present in eDirectory as a cluster node object, which is shown in Figure 2.1. This eDirectory object stores the node number and its IP address. This object is also used by eDirectory to create an alias to the NetWare Core Protocol (NCP) server object it represents.

Figure 2.1. All components of the cluster are represented as objects in the cluster eDirectory container.

[View full size image]

Cluster Resource

All applications in the cluster are represented by cluster resource objects. These cluster resource objects contain all important information about the resource as an eDirectory property. The most important properties of the cluster resource are listed here:

  • Load scripts: These determine how the resource will be loaded. These load scripts are written as bash shell scripts. All resources do have a cluster resource load script.

  • Unload scripts: In the unload script, the cluster resource is terminated properly. In the cluster resource load script, it is particularly important that other modules on which the resource depends are unloaded properly as well. Load scripts as well as unload scripts are discussed in Chapter 5, "Creating Cluster Resources."

  • Failover and failback modes: These modes determine what should happen when a node in the cluster on which the resource is currently active fails. The most important choice that has to be made here is that these can happen manually or automatically. More information on the failover and failback modes is found in Chapter 5.

Cluster Templates

It can be rather difficult to create a cluster resource object all by yourself. To make creation of cluster resources easier for the administrator, some cluster template objects exist. These templates contain all important properties for some often-used cluster resources.

Tip

After installing the latest support pack on your servers, new cluster resource templates that are provided with the support pack are not installed automatically. From a Linux box, use the /opt/novell/ncs/bin/ncs-configd.py-install_templates command to install them. You will want to do this if you're upgrading from NetWare to Linux; otherwise, you don't get the new Linux-specific cluster resource templates.

Cluster Volume Resources

In a NetWare environment, a shared disk that is used by Novell Cluster Services must contain Novell Storage Services (NSS) volumes. These volumes can be either cluster enabled or regular. Both of these volume types can fail over to another server when a node failure occurs. In a Linux environment, NSS is not the only file system that can be used. Traditional Linux file systems such as Reiser and ext3 can be used as well.

If an NSS cluster-enabled volume object is used, three eDirectory objects are created:

  • Cluster Volume: This is the "normal" volume object, used to represent the cluster-enabled volume.

  • Cluster Volume Resource: This object is similar to the resource objects used for clustered applications. It maintains the load script and unload script, as well as other clusterrelated properties of the volume.

  • Cluster Virtual Server: This is a virtual server object that hosts the clustered volume. This object is needed because a clustered volume is not bound to a physical server. Instead, the virtual server acts as a proxy to bind the cluster volume to a real server. The advantage of this solution is that the cluster volume will not go down if some node in the cluster goes down.

A cluster-enabled shared volume is not needed in all scenarios. If an NCP client reconnect to a volume is not needed and if no fixed server name is needed either, you can do without cluster-enabled shared disk volumes. When a node failure occurs with a volume that is not cluster-enabled, the cluster resource manager module will simply mount the volume on some other node and execute the associated load script.

Relation Between Edirectory Objects and Local Files on the Cluster Node

There is a relation between the objects in eDirectory and files that are kept locally on cluster nodes. The eDirectory information is cached in files in /etc/opt/novell/ncs (see Figure 2.2). In this directory, the load and unload scripts of all resources in the cluster are stored. You can locate these as the files with the .load and the .unload extensions. This directory also contains all eDirectory attributes as .xml and .conf files. For example, the protocol properties that are set at the cluster container level are stored in the gipc.conf file. As an administrator, you should not edit these files by hand. The nds-configd is a user-space Lightweight Directory Access Protocol (LDAP) client that is responsible for synchronizing data between eDirectory and the local files. Apart from the information that is in /etc/opt/novell/ncs, another important part of the cluster status and configuration information can be found in /_admin/Novell/Cluster. For example, the current state of the nodes in the cluster is saved here.

Figure 2.2. On Linux, all settings from eDirectory are cached in files in /etc/opt/novell/ncs and /_admin/Novell/Cluster.

[View full size image]

Cluster Modules

The Novell Cluster Services software consists of different functional parts. In this section we will discuss how these parts interact with each other. An overview of these parts is provided in Figure 2.3.

Figure 2.3. In Novell Cluster Services, different modules work together.

[View full size image]

Clstrlib

The first thing to notice from Figure 2.3 is that there is no real difference between the modules on NetWare and the modules on Linux. First, there is the cluster configuration library CLSTRLIB. This library provides the interface between Novell Cluster Services and eDirectory. It is the responsibility of the cluster configuration library running on the master node in the cluster to access all cluster-related configuration from eDirectory. The master node stores this information in its memory so that it is available for all other modules used by the cluster software. The master node is also responsible for transmitting this information to all other nodes in the cluster. As changes occur to the cluster configuration in eDirectory, CLSTRLIB accesses the changes, updates the information in its memory, and then transmits it to the other cluster nodes.

Gipc

The Group Interprocess Protocol (GIPC) tracks changes in cluster membership. It is responsible for handling events related to cluster membership, such as nodes joining the cluster and leaving the cluster. To do so, GIPC runs a set of helper protocols:

  • Panning: Filters messages to the current epoch.

  • Heartbeat: Responsible for generating heartbeat messages (see the next section in this chapter). The heartbeat information is layered over IP protocol 224 and can be monitored with the Ethereal packet decoder.

  • Sequencer: Responsible for sequencing of multicast and membership changes. This protocol is responsible for notifying all nodes in the cluster about changes in the cluster membership.

  • Membership: Maintains information about nodes that currently are a member of the cluster. Its main purpose is to reach an agreement between all nodes on the current cluster membership view.

  • Censustaker: Generates reports about unstable membership by monitoring heartbeat packages. This module will report when a node joins or leaves the cluster. This report is considered unstable until accepted by all other nodes in the cluster.

  • Group Anchor: Communicates group membership information to the upper modules in the cluster protocol stack.

Sbd

If a network connection between nodes in a cluster fails, a split brain scenario can occur if clusters use shared disk systems. Each side of the cluster could assume that nodes at the other side have failed and then try to take over the resources from the other nodes. This may cause data corruption. Novell Cluster Services uses a Split Brain Detector to prevent such situations. The SBD relies on a connection from all nodes in the cluster to a shared disk subsystem that works over a dedicated channel. On the shared storage device, a partition is created where all nodes record their view of the current state of the cluster. When changes in the cluster membership occur, these changes can be notified by means of this SBD partition. The following section of this chapter provides more details about the workings of the SBD.

Crm

The Cluster Resource Manager (CRM) is responsible for running resources on the correct nodes when changes in the cluster membership occur. This is the responsibility of the CRM on the master node, which receives an alert every time a node joins or leaves the cluster. The master node in the cluster tracks the state of all resources in the cluster and distributes that information to all other nodes in the cluster. If the master fails, the CRM running on another node in the cluster will take over this responsibility. The CRM monitors the following possible states of resources in the cluster:

  • Unassigned: The preferred nodes of the resource are not currently members of the cluster.

  • Offline: The resource is not active. This is the state you need to enter to be able to modify the properties of the resource.

  • Loading: The resource is currently being loaded.

  • Unloading: The resource is currently being unloaded.

  • Comatose: The resource could not process the complete load script before a timeout occurred.

  • Running: The resource is currently active.

  • Alert: The resource requires manual intervention.

  • NDS Sync: The resource is waiting to synchronize with its eDirectory properties.

  • Quorum wait: Currently there are not enough nodes in the cluster. The resource is waiting for the other nodes as defined in the quorum to become active.

Virtual Interface Architecture Link Layer

The Virtual Interface Architecture Link Layer (VLL) sits between modules in the cluster. It interfaces with the GIPC, SBD, and CRM modules. If the GIPC stops receiving information from another node in the cluster, it notifies the VLL. This module then contacts the SBD, which determines whether the node is really dead and then informs the CRM of its decision.

Cluster Volume Broker

The Cluster Volume Broker monitors NSS file system information for the cluster. If a change occurs, it will ensure that the change in the NSS file system is replicated to all other nodes in the cluster.

Cluster System Services

The Cluster System Services (CSS) module provides an application programming interface (API) that cluster-aware applications can use to enable distributed shared memory and distributed shared locking. Distributed shared memory is a technique that allows cluster-aware applications running across multiple servers to share access to the same data, as if the data were present on the same physical RAM chips. Distributed locking ensures that if one thread on one node gets a lock, another thread on another node can't get the same lock.

Cluster Services Management Agent

The Cluster Services Management Agent (CMA) acts as a proxy for the management applications. Several applications are available:

  • ConsoleOne: The legacy management application for NetWare environments. It cannot be used to manage NCS 1.8. It is, however, still needed to manage some applications in an NCS 1.8 environment. GroupWise is an example of such an application.

  • The Perl command-line interface (CLI): On Linux, some Perl utilities are available from the command line. These utilities are used from user space and interact with the data in the /_admin directory to modify and get information from the cluster.

  • iManager: The iManager management utility interacts with /_admin and CMA to modify cluster parameters.

The CMA interacts with the CRM to determine current information about the cluster and control the current state of cluster resources. It also performs SNMP management capabilities and Simple Mail Transfer Protocol (SMTP) alerting. To do its work, on Linux a kernel module exports all relevant management files to /_admin/Novell/Cluster. On NetWare the _ADMIN directory is used for the same purpose.

Pcluster

On NetWare, the Portal Cluster Agent (PCLUSTER) provides an interface to manage Novell Cluster Services from Remote Manager. This functionality is not available from Linux.


Heartbeats, Epoch Numbers, and the Split Brain Detector

One of the most important things that has to be arranged in a cluster is how the cluster will detect when a node fails. Novell Cluster Services uses heartbeats on the LAN and a Split Brain Detector on the shared storage device to do this. A heartbeat is a small package periodically sent over the LAN (and not over the storage network) by all nodes in the cluster. The master node sends a multicast to all slaves, and the slave nodes send unicast packets directly to the master node. The multicast packet will be picked up by all nodes listening to that particular multicast address. To avoid sending too much network traffic, the master node will reply with a unicast, which is a packet sent to that particular node directly. Related to the heartbeat, there is the tolerance rate, by default set to eight seconds. If a node fails to send heartbeat packages within the period defined as the tolerance rate, it will be removed from the cluster. This happens by crashing the other node; on NetWare the node that has failed to send a heartbeat package will be abended. We describe later in the text how this mechanism works.

A technique that is narrowly related to that is the epoch number that is written over the SAN to the Split Brain Detector partition. This is a small section on the shared storage device. When writing its epoch number, a node first compares the epoch numbers it wants to write to the other epoch numbers on the network. Each time a node joins or leaves the cluster, the epoch number is increased by one. A difference in epoch number may indicate that a critical situation has arisen and may result in casting of a node from the cluster. In the following discussion you will learn how a failed node is removed from the cluster via this mechanism.

Tip

The size of the Split Brain Detector partition cannot be specified when installing the cluster. It will, however, always be big enough to accommodate a 32-node cluster. Each of the nodes needs 512 bytes to communicate necessary status information to the other nodes. Because, however, a partition always needs to be one disk cylinder, a partition of about 8MB will be created. The remainder of this 8MB is used for the persistent cluster event logs. After the partition has been filled, the cluster will begin to reuse the oldest log entries.

Removing a Failing Slave Node

Several steps are taken when a slave node fails. These steps are described here:

1.
Each node in the cluster sends a heartbeat over the LAN. By default, this happens every second.

2.
At half-time of the preconfigured tolerance rate (which is 8 seconds by default), each node writes its epoch number to the Split Brain Detector on the shared storage device. Under normal circumstances the epoch numbers of all nodes in the cluster will be the same.

3.
The master node monitors the heartbeat packets and epoch numbers of all nodes in the cluster.

4.
If a slave node fails to send a heartbeat package within the period defined as the tolerance rate, the master node and the other slave nodes create a new cluster membership view. This new cluster membership view does not include the failing node.

5.
All nodes in the new cluster membership view update their epoch numbers by one and write this modification to the Split Brain Detector. At that moment, there are two cluster membership views: one with the node that failed to send heartbeats within the tolerance rate, and one with the other nodes. This situation is called a split brain.

6.
Novell Cluster Services now needs to determine what cluster membership view wins. It does so by voting between the two cluster membership views. There are three possibilities:

  • The cluster membership view that has the most nodes wins.

  • If there are equal nodes on both sides, the side with the view that contains the master node wins.

  • In a two-node cluster, the node that still has network connectivity wins.

7.
Based on the results of the vote, a special token is written by the winning nodes to the Split Brain Detector; this is the so-called poison pill. The losing node reads the poison pill and as a result crashes itself with an abend or kernel panic. Terminating the losing node is necessary because it ensures that the losing node cannot corrupt the new cluster.

8.
The new cluster migrates all resources previously associated with the failing node and thus guarantees uninterrupted services for end users.

Removing a Failed Master Node

The preceding scenario describes what should happen if a slave node fails. The situation is different when the master node fails because in this case a new master must be assigned as well. The following procedure describes what will happen. It should be noted that the process is more or less the same as what happens when a node that is not the master fails:

1. | After failure of the master node, a new cluster membership view is created. This new cluster membership view includes only the slave nodes. The master node maintains the old cluster membership view.
2. | The nodes in the new cluster membership view update their epoch number by one.
3. | Novell Cluster Services votes based on the new information.
4. | The cluster with all the remaining slave nodes wins the vote.
5. | The nodes in the winning cluster write a poison pill to the Split Brain Detector.
6. | The slave nodes vote who will be the new master. This voting mechanism is more or less random.
7. | The failed master node reads the special token and terminates itself.
8. | All resources assigned to the failed node are migrated to other nodes in the cluster.


Summary

In this chapter, we provided a short overview of the architecture and working of the Novell Cluster Services software. This overview should make it easier for you to analyze what exactly is happening if anything goes wrong. In the next chapter you will read about the design of a cluster based on Novell Cluster Services.


Chapter 3. Clustering Design

In this chapter we will provide you with all the functional and technical details you need to design your clustering solution. We will cover topics such as how many nodes to choose and how to configure eDirectory on your cluster. We will also give you some practical tips on how to set up clustering for applications that do not require shared storage.


Cluster Design Guidelines

In this section we will give you some general guidelines that will help you to design your clustering solution. You should really consider them as rules of thumb and not as the Law of Clustering. There are many parameters that can be different for your environment and that can make you decide to do it a little different.

How Many Nodes to Choose

Starting with NetWare 6, you can build a cluster with two nodes without the need to buy additional licenses. This is still so for Open Enterprise Server. It is therefore very tempting to implement just a two-node cluster. There is nothing wrong with building a two-node cluster per se, unless you do it for the wrong reasons. We have seen too many customers that implemented a two-node cluster to replace two existing servers and in that way tried to provide high availability.

Not only may it cause problems with the workload for a single-cluster node that will remain active in case of a failure, but also you will not have redundancy when you have to remove one node from the cluster; for example, to perform an upgrade. This is exactly the reason that Novell requires you to have at least three copies of an eDirectory partition to provide high availability for eDirectory itself (one master replica and two read/write replicas). If you had only two, whenever you had to bring a server down for maintenance or when a server failed, you would have no fault tolerance with only one single remaining replica. The same rule applies to building a cluster.

When looking at the number of cluster nodes you will need, it is important first to decide what services you want to run on these servers and what the server requirements are for these applications. Armed with that information, you can decide on the number of cluster nodes you will need. In most scenarios for up to six servers being replaced, it is a good idea to add one extra server to that needed number of cluster nodes. That way you can be certain that when a server fails, there are enough resources available on the remaining nodes that all resources run without any loss of performance. For this to work efficiently, create a failover matrix to define to what cluster node the resources will fail over in case of failure. How to create a failover matrix is explained later in this chapter.

For larger clusters from 6 up to 32 nodes, it will probably not be the case that you are replacing existing servers with a cluster solution. But even when this is the case, there will most likely be enough resources available in the cluster to run the services from a failed node, if you have created an efficient failover matrix.

When designing a cluster solution, you should also keep in mind that you can install two or more clusters in your environment. It might not be practical to build a large cluster of 16 nodes to run all your services. Even if your hardware and storage area network (SAN) scale up to this number of nodes, you should keep in mind that if you would ever need to do maintenance on the entire cluster, it affects your entire environment. Also, you cannot delegate part of the cluster administration to different administrators. This can especially be important in larger environments with a large number of servers, where, for example, a group of administrators can be in charge of application servers and another group handles file and print servers. To not have them interfere with each other, it could be better to have more than one cluster.

Using a Heartbeat LAN or Not

Novell Cluster Services (NCS) uses a heartbeat mechanism to detect whether all servers are available via the network. Besides this LAN heartbeat, there is also the Split Brain Detector (SBD) partition through which the status of servers is watched. It is possible to set up a dedicated LAN for the heartbeat channel. The benefit of this is that when the performance of the LAN drops, the heartbeat channel remains active without interruption. But this advantage is also a disadvantage: When the servers cannot contact each other for the heartbeat, the clients might also not be able to contact the servers. Especially when a network adapter on the client LAN fails and the heartbeat network remains active, the clients will lose their connection to the service that was handled by that network adapter, but the cluster node will stay alive. Figure 3.1 shows an example in which a network connection on the client LAN fails but the heartbeat remains active. The GroupWise post office running on the failing node is not moved to another node; thus, the clients will try to access it on the same host and will fail.

Figure 3.1. If a NIC fails, the cluster node stays alivenot good for your clients.

We advise against using a dedicated heartbeat LAN. It will prevent the previously described scenario, in which clients lose access to services, from happening. To be protected against communication failures due to a malfunctioning network adapter, it is better anyhow to set up network interface card (NIC) teaming. NIC teaming is described in the following text.

Use NIC Teaming

In Chapter 1, "Introduction to Clustering and High Availability," we commented that looking into your high-availability solution should not only be looking at Novell Cluster Services. You should look into high availability for more than your servers. Examples are network switches and your data center power supply. Something else that can help you improve the overall availability for your environment is to use NIC teaming. This technology can help you remove the single point of failure for your network connections. If a network board fails and another one remains available, it would no longer mean that the cluster heartbeat is interrupted and a failover has to be performed.

Two types of NIC teaming are available. The first type is implemented in the hardware such that the operating system is not aware of the redundant NIC and a special driver must be used to send an alert that an adapter has failed. The other type is one in which the operating system takes care of bundling the network adapters, called bonding, into one logical device with a vendor-specific driver. We will describe how to set up NIC teaming in Chapter 9, "Advanced Clustering Topics, Maintenance, and Troubleshooting."

In the most basic form of NIC teaming, shown in Figure 3.2, a server has two network adapters that are connected to the network. The switch will still be the single point of failure in this example. To also create a redundant configuration for this part of the network configuration, two switches can be used to link the network adapters to different switches. For that to work, the drivers must support the IEEE 802.3ad Link Aggregation Control Protocol (LACP), which most network card vendors such as Intel and HP support.

Figure 3.2. NIC teaming in its most basic form.

Choosing Storage Methods

In this section we will introduce the storage methods available for clustering. This is divided in three parts: the storage connection, the disk configuration, and the file system.

The Storage Connection

First of all, your cluster nodes need to have access to a shared storage medium. The choices you have are widespread, but we will discuss the three most common ones: SCSI (Small Computer System Interface), Fibre Channel, and iSCSI. The first technology is really a direct connection method, and the two others will form a storage area network.

Using Scsi For Your Shared Storage

For a long time SCSI was the most used technology for attaching storage and other equipment to computers. With SCSI, disks and tape devices are attached to a SCSI bus that transports data blocks from one device to another. Figure 3.3 shows an example of a cluster with shared SCSI storage.

Figure 3.3. A shared SCSI cluster in its most basic form.

These types of clusters are simple in their nature. The two servers both contain a SCSI adapter that is set to a different SCSI ID, 6 and 7 in this example. The SCSI disk takes part of the SCSI bus too and is set to SCSI ID 0. Other devices that would be on the same SCSI bus would also be configured with a unique device ID.

Many hardware vendors sell servers in a clustered configuration with the servers and storage built into one rack enclosure or server casing. We use the phrase "cluster in a box" for such equipment.

SCSI is a technology that can be used for a two-node cluster. With the special hardware configurations for a shared SCSI cluster described before that are not too expensive, it is a good choice if you want to implement high availability in a small organization or at the departmental level. A two-node cluster should be deployed to replace the load of one single server and give the services from that server a higher availability.

Building a San With Fibre Channel or Iscsi

A storage area network is really what the name says: a network of computers and storage devices that access storage in a special-purpose network. Such a network can be built with a handful of technologies, but we will look at the two most commonly used ones: Fibre Channel and iSCSI. The latter is discussed in great detail in Chapter 7, "iSCSI," so we will not go into too much detail for iSCSI here, and we will focus on what a SAN is and how Fibre Channel can be used to set it up. Figure 3.4 shows a basic example of what is needed to build a SAN with Fibre Channel equipment.

Figure 3.4. A basic configuration of a Fibre Channel storage area network.

Every server needs to be equipped with a Fibre Channel Host Bus Adapter (HBA). From the server adapters a fiber-optic cable runs to a Fibre Channel switch. And last but not least, a disk system is attached to the same Fibre Channel switch.

The way that the disks can be used from the operating systems running on the servers is that the SAN provides LUNs (Logical Units) to the servers. Each LUN can be used as if it were one local disk in a server. Management applications such as iManager and NSSMU will display a LUN as a device, just as they would do for a local physical disk.

If you look at Figure 3.4 with high availability in mind, you'll notice that the servers are redundant in that picture, but the SAN has several single points of failure. Therefore it is possible to add some redundancy into the SAN. Each server can be equipped with two HBAs, each connected to a separate switch. And the storage can be connected with two cables into these separate switches. In such a more redundant scenario, a switch or an HBA can fail without interrupting the service of the SAN. When you have such a setup in place, it is important that some sort of management algorithm be included in the SAN to take care of the multiple paths that become available. This is known as multipath support. Multipath is implemented in Novell Storage Services (NSS) but is also included in the drivers from many HBA manufacturers, such as QLogic, Hewlett-Packard, Emulex, and LSI Logic.

Fibre Channel technology can be used to create the largest clusters possible. It provides the fastest data throughput of all technologies discussed here. We are not sure whether we should call it a disadvantage, but for setting up a Fibre Channel SAN special technical skills are needed. Many customers let their hardware resellers set up the SAN for them to the point where LUNs are available to use from the clustering software. You may compare the technical expertise level needed to configure a SAN with that of configuring a switched ethernet network with Virtual LANs (VLANs) and spanning tree.

The Disk Configuration

After you have set up a physical connection from every server to a storage device, it is important to look at how the disks are being used inside that device. This can range from a single disk to using hardware RAID (redundant array of independent disks) or software mirroring configurations.

Single-Disk Configurations

These types of configurations are valid only for really unimportant data that can be missed for a day or two. You could be thinking of an archive that you want to keep online. In the old days there were technologies to keep data available on tape or optical devices with an easy mechanism to collect the data when needed. This was called near-line storage. With the availability of inexpensive large IDE disks, it has become possible to store such data online for a fraction of the cost that is required for either a tape library or expensive SCSI disks. In your clustered environment you can use a disk enclosure with 500MB IDE disks or an iSCSI target server with those disks to store data that is available through the cluster. But even then we advise you to use one of the technologies described hereafter to create redundancy for your disks. Especially with these not-too-expensive IDE disks, the time to restore data can be brought to zero for only a few hundred dollars.

Redundant Disk Configurations With Hardware

The most widely used technology for disk redundancy and fault tolerance is RAID. Many levels of this technology are available, the most common being RAID 1 and RAID 5. But RAID 0+1 or RAID 10 is sometimes used. We will first explain some terminology here about RAID that we will use in the remainder of this section. An array is a set of combined disks. Two or more disks are grouped together to provide performance improvement or fault tolerance, and such a group is called an array. As for the operating system that works with that array, we call that a logical disk.

Note

First of all, a warning here about RAID 0: With that technology there really is no redundancy. There is performance improvement because the data is striped across multiple disks. But if one disk in a RAID 0 array fails, the entire array becomes unavailable.

RAID 1 mirrors disk blocks from one disk to a another disk. This means that every disk block being written to disk is also written to the second disk. The operating system connecting to the array sees one logical disk. It is the hardware RAID controller that takes care of the mirroring. In this scenario one disk could fail and the logical disk is still available to the operating system. RAID 1 improves the performance for read operations because data can be read from either disk in the mirror. This type of redundancy is the most expensive in terms of disk usage because it requires a double amount of disk capacity that is effectively needed.

RAID 0+1 and RAID 10 are both combinations of RAID 0 and RAID 1. With RAID 0+1 the RAID controller combines physical disks into a stripe set as a virtual logical disk. It does that for a minimum of two physical disks and also for two other physical disks. It then uses those two virtual logical disks, that are both striped, internally and mirrors them into one logical disk for the operating system. In a RAID 10 configuration it works the other way around: A minimum of two physical disks are first mirrored into a virtual logical disk, and the same happens for two other physical disks, after which the RAID controller stripes data over the virtual disks that are created. The effect of using these technologies is that they improve performance compared to regular RAID 1 mirroring. RAID 10 is your better choice over RAID 0+1 because it uses mirroring as the first defense against disk failures, whereas RAID 0+1 first stripes the disks and then uses mirroring only internally for the combined virtual disks.

In a RAID 5 configuration, data is striped across multiple disks and redundancy information is stored on the disks in the array. The way this works is that when blocks are written to the physical disks, a parity block is calculated for a combination of blocks of physical devices and written to one of the disks in the array. There is no single disk that holds all the parity information; it is striped across the disks in the array. In this configuration one of the disks in the array can fail and still all data is available because it can be reconstructed with the parity information from the remaining disks. Only when more than one disk fails does the entire array become unavailable. The great advantage of RAID 5 is that the extra disk capacity that is needed is kept to a minimum. For a combination of four disks, only one extra disk is needed. Buying five 72GB disks will provide you with a net 288GB disk capacity. The main disadvantage of RAID 5 is that it has a lower performance compared to RAID 1 because of the additional overhead of the striping process, whereas RAID 1 just writes the data to two disks without having to think about that.

An important feature of RAID configurations is that they support hot spare disks and sometimes also hot pluggable disks. With the first feature you can have a disk on standby that can be used to replace a disk immediately in case of a failure. If it supports hot-pluggable disks, you can add disks to the server while it is up. Depending on the RAID controller, you can even expand the array online and add the new segments to the operating system online. These two features are important to look out for when evaluating RAID controllers for your environment.

Redundant Disk Configurations With Software

Besides redundancy that comes as hardware in your servers, it is also possible to use the operating system to create redundancy for your disks. This can be striping to improve performance (RAID 0), mirroring of disks or partitions (RAID 1), or even software striping for fault tolerance (RAID 5). The only solution we think could be used in any professional setup is software mirroring. Novell Storage Services supports this at the partition level. It can be used with physical disks but it is also a possible solution to mirror disks that are used with iSCSI.

On Linux there is also the possibility to use software RAID. Independent disk devices can be combined into a single device, for example, called /dev/md0, to provide fault tolerance. Check the manual pages for the commands raidtab, raidstart, and mkraid to see how to set up a software RAID configuration, or check the documentation on RAID at the Linux Documentation Project at the following URL: www.tldp.org/HOWTO/Software-RAIDHOWTO. html.

Another solution to create redundancy for disks in Linux is the Distributed Replicated Block Device (DRBD). This is a RAID 1 solution running over the network. In this configuration one node has read-write access to the DRBD device and the other node does not. This technology is described in detail in Chapter 11, "Using SUSE Linux Enterprise Server Clustering Options," where it is used for Heartbeat, a high-availability solution for Linux.

If you can afford it, always use hardware redundancy with RAID 1 or RAID 5. You will get better performance because the RAID controller contains its own processor and cache memory to do its job. Also the management and alerting for hardware RAID are better than what you would get in any operating system.

The File System

With the storage connections in place and the disks configured for high availability, it is time to look at the last part of the storage component of a cluster solution: the file system.

Not all file systems are cluster aware and thus not all can be used with Novell Cluster Services. For NetWare there is nothing more to choose from than NSS, but not so for Linux.

When we first look at NetWare, we can use NSS as a file system in a cluster because it contains a feature called Multiple Server Access Prevention (MSAP). A node that activates a pool on the shared disk writes a flag to that disk to indicate that the pool is in use by that server. Other servers that have access to the shared disk will see that flag set and will not activate the pool. This prevents corruption that could occur when two servers were writing data to the same data area at the same time.

For Linux, NCS also works with NSS, but it can also support other file systems. Ext3 and ReiserFS are not cluster aware by nature; but they can be mounted on a server and whenever a failure occurs they can be mounted on another node in the cluster.

There are also true Linux cluster-aware file systems available. Novell Cluster Services supports the Oracle Cluster File System version 2 (OCFSv2), but other file systems that can be used for building a Linux cluster are RedHat's Global File System and Polyserve's symmetric cluster file system.

For the Open Enterprise Server environment your best choice is to use NSS, or alternatively when you want to cluster-enable Linux applications with OES, you could use Ext3 or ReiserFS.

In Chapter 8, "Cluster-Enabled Applications," we explain what file systems to use for different types of applications that you can run in your Open Enterprise Server Linux environment.

Mirror the Split Brain Detector Partition

For a cluster with shared storage, a small part of the shared disk, or maybe a special LUN on the SAN that you assign for this purpose, will be used as the Split Brain Detector partition. All cluster nodes write data to this disk to report about their status. If a cluster node does not respond to network heartbeat packets, it can be given a poison pill by the other nodes in the cluster and remove itself from the cluster. That poison pill is nothing more than a flag that is set on the Split Brain Detector partition.

If access to the Split Brain Detector device is interrupted, a node will take itself out of the cluster. The reason for this is that it assumes that this failure has also interrupted access to the other shared data and thus would impact the functionality of the clustered applications.

The other nodes in the cluster will then load the applications. Or that is what should happen. But let us look into what happens if all nodes lose access to the SBD partition. All of them will assume that there is a problem with the shared storage and will take themselves out of the cluster.

This situation can occur because of a hardware failure, either because a disk fails or because a Fibre Channel device, such as a switch, fails or an administrator unplugs a SAN cable. In an iSCSI environment it can happen if the iSCSI target that holds the SBD partition is stopped.

The solution to this problem is to mirror the SBD partition. During installation of Novell Cluster Services, you can select to configure mirroring if a second device that can hold the SBD partition is available. It is also possible to re-create the SBD partition and to select mirroring at that time. How to do that is discussed in Chapter 9.

Selecting Applications to Run in a Cluster

The way that Novell Cluster Services works is that when a node fails, another node can load the applications that the failed node was running at that time. We call this type of clustering a failover cluster. Other cluster technologies offer a seamless uninterrupted continuation of an application because it is already active on more than one node at a time. These types of clusters are more complex and more expensive and are not available for Open Enterprise Server (OES). So for OES we work with applications that perform a failover. This also has some implications for the type of applications that can be run in a cluster.

Let us first look at a sample application to see how the server and clients behave in a cluster. An Apache web server is running on node 1 and reads its document directory from the shared disk. A user has just loaded a web page from the server and is reading through it. At that time node 1 fails and node 2 activates the shared storage and loads the Apache web server with the exact same configuration and IP address. If the user is finished reading the web page and the server has become online on node 2 in the meantime, the next page the user accesses will be served from node 2 and availability will not be interrupted.

This scheme works for most applications that we can run in the cluster. For example, a user that is accessing GroupWise will not notice if a failover is performed. But this will not be true for all types of applications. It depends on the behavior of the application whether the clients will survive a failover. If the web server that we introduced earlier is running a web-based application for a shopping-cart system with server-side session information about shopping baskets and customer information for logged-in customers, all that information, which is very often in memory, will be lost when the node fails, and the customers will have to reestablish their sessions.

Other types of applications that may not survive a failover are databases that run on cluster nodes. These will very likely also contain session information about users that are logged in, and the client applications may not support auto-reconnecting to the server.

There is no extensive list of applications that can be used as a reference for cluster-enabled applications. But a good starting point is the overview from Novell's documentation website of applications that can be configured for a cluster. The applications known to work well in a Novell cluster are listed here:

  • Apache web server

  • BorderManager

  • DHCP

  • DNS

  • exteNd application server

  • FTP server

  • GroupWise

  • iFolder

  • iPrint

  • MySQL

  • Native file access methods for NetWare (AFP, NFS, CIFS)

  • NetStorage

  • Tomcat application server

  • User data (through NSS)

  • ZENworks for Desktops

  • ZENworks for Servers

For all applications not included in the preceding list, it is up to you as the administrator to test whether they will work in a clustered environment. You can do that on the test environment that you already are using for your cluster environment. If you do not already have cluster hardware and you want to evaluate whether your applications would work in a cluster, you can also build a cluster in a virtual environment with VMware. How to build such an environment is explained in detail in Chapter 4, "Installation and Configuration."

Tip

For applications that are developed in-house, you can sometimes make them cluster aware when programmers are made aware of how they could handle the application in a clusterfor example, by keeping application state information and client session information on disk instead of in memory. The disk information will be transferred to another node, and what was in memory will be lost.

eDirectory Cluster Guidelines

Novell Cluster Services uses eDirectory as its repository for configuration information, and eDirectory also is the main database where information is stored about all resources that are configured for access to the cluster. Most applications that run on the cluster also use eDirectory to store their configuration and to control access to its resources.

The cluster configuration is stored in the cluster container. This container object itself contains general configuration information for the cluster. And the container holds all objects used by the cluster: cluster node objects, cluster resources, and cluster volumes. Every server in the cluster needs access to these objects in a local replica. It would also not make much sense if the cluster had to read information about itself to operate, when that information comes from outside the cluster. If the external resource was not available, the cluster would not be able to operate. Therefore, it is a good practice to create a partition for the cluster container and place a replica of this partition on each node. There is no need to place the master replica of the partition inside the cluster. If you have a dedicated server holding all master replicas, you can place this master replica also on that server.

Other eDirectory design concerns are for the partitions where your user and application objects reside. Do you place a replica of these partitions on the cluster or not? We suggest that you always place a replica of these partitions on every cluster node that will be accessed by the users in that partition. If you do not have a replica of the user partition, the cluster node will create external references anyhow, so it makes more sense to have the real objects available locally. An external reference is a pointer that a server creates to an eDirectory object in case that server does not hold a copy of that object in a local replica. The external reference contains only minimal information about the object. The original object is also backlinked to the external reference, and thus these links require maintenance and are involved in deleting and renaming objects that have external references.

One important rule here is that you must have an eDirectory environment with version 8.6 or higher. The improvements Novell has implemented in that version are so significant that the great scalability of eDirectory has improved even more. This is vital for a well-functioning and scalable cluster.

With older versions of NDS, you really had to be careful where to place replicas. A maximum of 10 replicas was advised. With eDirectory this limit has been expanded. But even then it was not wise to have a large number of replicas for a partition. Synchronization cycles still had to contact a relatively large number of severs. The mechanism of transitive synchronization that was implemented with NDS version 7 and thus also in eDirectory has improved the synchronization cycle, but there still was a large amount of traffic that had to be transported. With the original synchronization process every server had to contact every other server in a synchronization cycle. With transitive synchronization all servers maintain a table where they keep track of servers that are already updated by other servers. Therefore, they do not all have to contact every other server if they are already updated by another server. With eDirectory 8.6 this transitive synchronization process has improved even more, and there is also multithreaded outbound synchronization. Because of this, you can have more than 10 replicas of your partitions, and placing them on the cluster does not bring too much overhead for your cluster nodes.

Tip

Coming from our practical experience, we advise you to keep the number of replicas as low as possible and to 10 as a maximum. If you ever had to troubleshoot eDirectory problems in a really large environment, you would find out how much time it costs to validate information in a replica ring with more than 20 replicas.

When designing your eDirectory environment, also keep in mind that there are applications that require that eDirectory information be available in a local replica and that having the information available locally will improve the performance of that application. You can think of eDirectory-dependent applications such as domain name services (DNS) and Dynamic Host Configuration Protocol (DHCP) on NetWare and also the Service Location Protocol (SLP) directory agent. For these types of applications, it is a good idea to create separate containers that can be partitioned off to store objects for the specific application. That partition can then be replicated to the servers where the data is needed.


Creating a Failover Matrix

When a failover occurs, your cluster resources will be loaded on another node in the cluster. The order in which the resource will fail over to other nodes is predefined in the cluster resource. By default, the list of servers to which the resource will fail over is populated with all the nodes in the cluster, in alphabetical order. This means that a resource loaded on OESNW1 will fail over to OESNW2. When OESNW2 is not available, it is loaded on OESNW3, and so on. When OESNW2 holds the resource and crashes and OESNW1 is not online, it will also go to OESNW3. If OESNW1 is online at the time of the OESNW2 crash, the resource will be loaded on OESNW1.

With the default scenario all resources will be moved to the same server. Therefore, it is a good idea to set up a failover matrix to design the failover paths. With such a matrix a basic level of load balancing is possible. Table 3.1 contains an example of a three-node cluster with six resources that will all follow a different path when the server they are assigned to fails. You implement these failover paths by assigning a different load order, or priority, for each cluster resource.

Table 3.1. An Example of a Cluster Failover Matrix

Resource

Prio 1

Prio 2

Prio 3

Data volume

OESNW1

OESNW3

OESNW2

Apache/MySQL

OESNW2

OESNW1

OESNW3

GroupWise

OESNW3

OESNW2

OESNW1

DNS/DHCP

OESNW1

OESNW2

OESNW2

iFolder

OESNW2

OESNW3

OESNW3

CIFS Server

OESNW3

OESNW1

OESNW1


Application-Specific Design Guidelines

In this section we will introduce some guidelines for installing applications on your cluster. For several applications we have added detailed guidelines in Chapter 8for example, for GroupWise and iFolder. But whatever application you have to install, these guidelines will make them more robust and easier to manage.

Install Applications on the Shared Disk

In a clustered environment, there is a central storage facility to which all the cluster nodes have access. The servers can work with the data if they have the clustered resource assigned to them. An example of this is the GroupWise domain and post office directories that are stored on the external shared storage to which the MTA and POA on the cluster node need to have access.

It is obvious that the shared data is not on a local disk on one of the servers. Therefore, why should the application be stored on all local disks of all cluster nodes? This is what you will find in much of the clustering documentation: Install the application on every cluster node to make sure that every server can load that application and work with the shared data. This approach creates extra work, because you will have to install it the same number of times as you have cluster nodes. Additionally, for every update, you have to update all the servers in the cluster.

An easier and more efficient approach is to install the application on the external shared storage. That way, there is only one installation required, and if the application needs updating, only one update does the job for the entire cluster. Another advantage of this approach is that in case a server's hardware crashes, you only have to replace the server with a new NetWare or Linux server and add that to the cluster. The rest will come from the shared disk.

As an example, we'll discuss what this looks like for a GroupWise environment. In your cluster, there will be a storage pool with a volume where the GroupWise domain and post office are located. Let us call that volume GROUPWISE. On the GROUPWISE volume, you create a GRPWISE folder where all of GroupWise's files are stored. You will then create a SYSTEM directory in the root of the GROUPWISE volume, where you install the necessary agents and configuration files. This system directory will also contain the NetWare Control File (NCF) script files to start and stop GroupWise. The directory structure would look like the following list (we left out a few default directories to keep it simple):

GROUPWISE:\SYSTEM

GROUPWISE:\GRPWISE\DOM

GROUPWISE:\GRPWISE\PO

One important item to add to your NCF script files is a search mapping to the new SYSTEM directory on the GROUPWISE volume. Otherwise, the application will not have access to modules that it would have to auto-load.

Execute NCF and Script Files from the Load and Unload Scripts

The configuration of what happens when a failover is performed or when a resource is migrated is defined in the load and unload scripts of the cluster resource. In these scripts, you enter the commands to start and stop the correct modules, add IP addresses, and so on.

Our tip here is to not add all those commands to load and unload scripts but to put everything into NCF files or Linux script files and call those files from the scripts. This has the following advantages:

  • You can set up delegated administration for your application and the cluster. For example, if you start GroupWise from an NCF file or a Linux script, there is no need for the GroupWise administrator to have rights on the cluster object, and the GroupWise administrator can use his own scripts to stop and start GroupWise, without having to work with cluster management software.

  • You do not have to bring the cluster resource offline and online when making changes to the load/unload commands, which you must do when they are in the load and unload script.

  • You cannot run into the problem in which the commands do not fit into the text box available for script commands. The maximum number of characters for the load and unload scripts is 600 total.

The best place to store these load and unload script files is on the shared storage. That way you do not have to manage files on all individual cluster nodes. You can manage one centrally stored file, making it easier to update the file when required.

It is the best option for the default commands that are added to your cluster volume resource to stay there. Do not move them into the NCF file or Linux script because they will be specific to the cluster and not to the application, and the cluster will manage them for you. For example, when you change the IP address of a cluster resource in ConsoleOne or iManager, these management applications also automatically modify the Add Secondary IP Address statement in the load and unload scripts.


Clustering Without Shared Storage

Most clustered applications work with files on the file system and therefore require access to shared storage. Examples of such applications can be found in Chapter 8GroupWise, iFolder, and Samba. But there are also applications that do not require access to files on the file system. So there really is no need for them to run on a cluster with shared storage. In this section we will discuss Novell applications that can run in a cluster without the need for setting up shared storage.

This means that you can use an existing cluster where these applications can be installed without configuring extra storage pools. But it also means that you can set up a basic twonode cluster without any shared storage at all to provide high availability for these types of applications. And this is free because Open Enterprise Server comes with a license for a twocluster node.

An example of a service that can be set up without shared storage is Lightweight Directory Access Protocol (LDAP). For LDAP the server reads its configuration from eDirectory, which, if set up correctly, is replicated to all servers. The only thing needed to set up LDAP for high availability is to configure the two servers in your cluster for LDAP and to set up a secondary IP address for use with this service.

We have chosen LDAP as an example, but there are more applications that just like LDAP do not rely on shared storage for their operation. DNS, DHCP, and SLP are examples of such applications. They work with data that is stored in eDirectory or on the local server hard disk. Chapter 5, "Creating Clustered Resources," describes how DHCP can be configured on both NetWare and Linux.

Prepare the Cluster

Setting up a cluster without shared storage is done in the same way as when shared storage is involved. We will provide the basic steps here. First, plan which two servers you want to use for this cluster and what the cluster's IP address will be. This must be an address in the same subnet that your servers are already in. Also, decide where the cluster container object must be placed in your tree.

The actual installation for a NetWare cluster is performed with the Deployment Manager (NWDEPLOY.EXE), located at the root of your NetWare 6.5 operating system CD. This utility contains the option Install/Upgrade Cluster in the Post Install tasks on the left, for installing a cluster. For Linux you start YaST to set up the cluster nodes, as described in Chapter 4.

During installation for NetWare, you will be prompted with a question, asking whether your server has shared storage. The answer is set to No by default if no shared storage is found. In a cluster scenario where you do have shared storage, this is the point when you would cancel the installation to find out what is wrong. However, for this cluster select Yes to continue. The cluster will be installed, and if you have selected to do so, the software will also be started automatically. For Linux you just leave the field where to configure the shared storage empty.

You now have a cluster you can use to create high availability for networking services. We will discuss how to put LDAP into this cluster next.

Note

The network services that can be clustered in this way do not rely on shared storage. They do rely on information stored in eDirectory. Therefore, make sure that both cluster nodes contain a replica of the partition where the objects needed for the clustered application are stored. In the case of SLP, this would be the partition where the SLP scope is stored; for DNS/DHCP, this would be the partition with the zone and address range information; and for LDAP, it would be the partitions with the objects that are used by the LDAP application. For the latter, you can also use filtered replicas to keep the eDirectory working efficiently, while still having all the needed objects stored in a local replica on the server.

The first step is to set up LDAP the way it needs to work and to store eDirectory replicas on both servers. Create a setup just as you would do for any normal server, where you would be running LDAP for your application. For this typical scenario, you may want to set up one LDAP group and make the two LDAP servers a member of that group to keep the part that is configured through a group consistent throughout the LDAP environment. However, keep in mind that there are also LDAP serverspecific settings you must set identically for both servers.

When you are finished with the LDAP configuration, create a new cluster resource. Do that from either iManager or ConsoleOne. Figure 3.5 shows an example of creating the resource in iManager. Do not use any of the available templates. Just create a new blank resource.

Figure 3.5. For an application without shared storage, create a simple cluster resource.

In the cluster resource for your LDAP server, configure the load script to add the secondary IP address to the server when loading the resource. Also, configure the unload script to delete the address when bringing the resource offline or migrating it to the other node in the cluster. Here's an example in which we used the address 192.168.1.120:

LDAP Resource Load Script | LDAP Resource Unload Script
add secondary ip address 192.168.1.120 | del secondary ip address 192.168.1.120

After setting up the LDAP server in the cluster, it is ready to be used. There is no need to reset the LDAP server or to load or unload the NLDAP.NLM. The LDAP server automatically binds to the new secondary IP address as soon as it becomes available on the server.

Tip

If you need to set up a secure LDAP server, it might be necessary to use the same server certificate on each cluster node for the newly added secondary IP address. How to add the same certificate to two servers is explained in Chapter 9.


Chapter 4. Installation and Configuration

Before you can use Novell Cluster Services (NCS), you have to install it. In this chapter you will find everything about installation of Novell Cluster Services.


Installing NCS on OES NetWare

To install NCS on OES (Open Enterprise Server) NetWare, you can use Deployment Manager or you can start the installation from iManager. Deployment Manager can be found on the OES NetWare installation CD. Unfortunately, up to OES NetWare SP1, this tool can be used only from a Windows workstation. When you log in to iManager, select the cluster installation from the Install and Upgrade task. The installation in iManager will also need access to the OES installation CD-ROM and will start the same installation program. Therefore, we will discuss only using the Deployment Manager.

Note

In this procedure, it is presumed that all necessary hardware is already in place. This hardware includes two NetWare servers or at least one NetWare server for installation in a mixed environment with Linux running on the other node, and a shared disk subsystem connected to all nodes (although this shared disk subsystem is not strictly required in all scenarios). For the purpose of this book, no difference is made between a shared disk subsystem based on iSCSI and a SAN (storage area network) solution based on Fibre Channel.

Before you start the installation, make sure that all the requirements have been met as well. For NetWare these include the following:

  • You must have a minimum of one NetWare server and one Linux server in a mixed environment, or two NetWare servers in a single-OS environment.

  • You must have at least 512MB of memory on all servers; 1024MB or more is recommended.

  • You must have at least one local disk device on each server for the SYS: volume.

  • All servers need to be configured with IP and must be in the same subnet.

  • All servers must be in the same eDirectory tree.

  • An IP address must be available for the cluster object, and one IP address needs to be available for each clustered resource as well.

  • If you want to use a shared storage system, the shared disk has to be set up properly, and at least one cylinder must be free for installation of the Split Brain Detector (SBD) partition.

When all prerequisites are met, you can start the installation as described here:

1.
Insert the OES NetWare installation CD into the CD drive of your computer. This will automatically start Deployment Manager. Or alternatively log in to iManager to start the cluster installation.

2.
Under Post-Install Tasks, click Install/Upgrade Cluster, and next click the link Install or Upgrade a Cluster. This will start the Java installation utility.

3.
Click Next in the opening screen of the installation utility. Next, select Create New Cluster to start the installation. In this screen, you will see the Skip the File Copy option as well (see Figure 4.1). This option is checked by default: The reason is that all files needed to create the cluster are copied to the NetWare server by default upon installation. You may uncheck this option, if you want to copy all files again, but normally this is not needed. Click Next to continue.

Figure 4.1. When installing the cluster, you don't need to copy all files to the server again.

[View full size image]

4.
In the next screen, as shown in Figure 4.2, some information needs to be entered to create the cluster objects. To start with, enter the name of the cluster object. Next, enter the name of the tree where you want to install the cluster; finally, enter the name of the container in eDirectory where you want to create the cluster. In this container, a special-purpose container will be created in which all cluster objects will be installed. You can use the Browse button to enter the names of the tree and context, or you can enter them manually. After entering the required information, click Next to continue.

Figure 4.2. To create the cluster object in eDirectory, the cluster object name and the location where it has to be created in eDirectory have to be provided.

[View full size image]

5.
Now select all NetWare servers you want to add to the cluster (see Figure 4.3). Linux servers cannot be added this way. After adding all NetWare servers, click Next to continue.

Figure 4.3. From the Cluster Installation utility that runs from Deployment Manager, only NetWare servers can be added to the cluster.

[View full size image]

6.
Next you have to enter a unique IP address for the cluster. This address must be in the same subnet as all the servers you are adding as nodes to the cluster. This IP address will be added to the master IP address resource. This object is created automatically and will be maintained by the master node in the cluster, regardless of which node that is at a given point. Enter the IP address and then click Next to continue.

7.
Now you have to specify whether the cluster has shared media (see Figure 4.4). If it does, the Split Brain Detector partition is created on this media. It is, however, possible to create the cluster without a shared media; you can find more information about this in Chapter 3, "Clustering Design." If you want to use shared media, for greater fault tolerance you can choose to mirror the shared media. Of course, you can do this only if mirroring is supported by the cluster hardware. Next select the storage device you want to use for the shared media, and click Next to continue.

Figure 4.4. In most situations, it is preferable to create the cluster with access to shared media.

[View full size image]

Tip

Be sure that access on the shared media is configured on all servers you want to add to the cluster. If one of the servers is not able to access the shared media at this point, you will be not able to select the shared media as indicated in step 7 of the procedure.

8.
In the following screen you can indicate whether you like the cluster software to be started automatically on startup. In most situations, it makes sense to select the option Yes here; if, however, some software has to be started and fully initiated before the cluster can be started, you can specify that the software shouldn't be started automatically in this screen. After you click Next, all eDirectory objects related to the cluster are created and the cluster software is started. If the cluster software starts successfully, this will be indicated in the Cluster Membership Monitor that is started on the servers in the cluster automatically. You can see an example of this in Figure 4.5.

Figure 4.5. If Cluster Membership Monitor comes up and all nodes appear as up, the installation was successful.

[View full size image]

Exploring eDirectory After Cluster Installation

Installation of Novell Cluster Services adds some objects into eDirectory (see Figure 4.6). We will now provide a short overview of all added items:

Figure 4.6. After installation of NCS in your tree, some cluster-related objects are added to the tree.

[View full size image]

  • The cluster container: When adding a cluster to your eDirectory tree, a cluster container is created for each cluster you add. In this container you'll find all cluster-related objects.

  • The Master_IP_Address_Resource: Upon creation of the cluster object, you have specified an IP address for the cluster. This IP address is added to the Master_IP_Address_Resource.

  • The cluster node object: For each server added to the cluster, a cluster node object is created.

  • Cluster template objects: Some template objects are created as well. These template objects make it easier for you to create cluster resources as described later in this chapter and in Chapter 6, "Cluster Management."

Apart from these default objects, more cluster-related objects are added to the tree when cluster resources are created.

Adding a NetWare Node to a NetWare Cluster

Adding an OES NetWare node to a NetWare cluster is easy: Just use Deployment Manager to perform this task. The procedure is very similar to the initial installation of the clustering software:

1.
Start Deployment Manager from the OES NetWare installation CD, and select Post-Install Tasks, Install/Upgrade Cluster. Next click Install or Upgrade a Cluster to start the installation program. In the opening screen, click Next to continue.

2.
Select Add New Nodes to Existing Cluster and click Next to proceed.

3.
Enter the name of the cluster object to which the new node needs to be added, and select the eDirectory tree and context where this object can be found. Then click Next. The configuration of the existing cluster object is now read from eDirectory.

4.
Enter the name of the server you want to add to the cluster. Then click Add to Cluster to add eDirectory to the cluster. Click Next to proceed.

5.
Under Start Options, select Yes to specify that you want the clustering software to be started automatically on the new node, and click Next to go on.

6.
To install the third node in the cluster, you need an NCS license as shown in Figure 4.7. This license is in a license envelope. Now select the license you want to use and click Next to install it into eDirectory.

Figure 4.7. You need to use a license for each additional node you want to install.

[View full size image]

Tip

You can skip license installation at this moment and install the licenses later if you like.


Installing NCS on Linux

One of the beautiful things of NCS is that it is cross-platform. An NCS cluster can consist of NetWare as well as Linux nodes, and in that case some resources can even be migrated across these nodes in the multiplatform cluster. Of course, you can't migrate a typical NetWare utility, running as an NLM (NetWare Loadable Module), to the Linux platform, because the necessary platform support is missing. If you are working, however, with a platform-independent resource like an NSS pool, you can migrate the resource across the different nodes in the cluster. The installation procedure of NCS on OES Linux however, is completely different:

1.
On the OES Linux server, start YaST, enter the administrator password, and select System, Novell Cluster Services (NCS). You will now see a message indicating that Novell Cluster Services needs NSS to be installed as well (see Figure 4.8). You must click Yes to continue and install both products; otherwise, you cannot proceed with the NCS installation.

Figure 4.8. On OES Linux, you need to install NSS if you want to install NCS.

[View full size image]

2.
In the window that pops up, click Continue to proceed. Insert all CDs the installer asks for.

3.
In the Novell Storage Services LDAP Server Configuration screen, specify whether you want to access a local directory server or a remote directory server for LDAP access. If your server has a replica, select Local System; otherwise, select Remote System to contact an LDAP server. Enter the Admin password and click Next to continue.

4.
Now enter a unique object name for the NSS admin volume. In most cases, you can accept the default value. Then click Next to proceed. The NSS settings are now written to eDirectory and the installer continues installing the NCS software. In the next screen, click Continue to proceed. Then insert all CDs asked for.

Note

The NSS Admin volume is used to keep administrative information for the NSS volumes on a server. It is always required if NSS is used.

Tip

Make sure that you insert the right CD. OES Linux consists of two different sets of CDs. First there are the OES CDs that start from number one of the CD set. Later in the installation, the installer prompts for SUSE Linux Enterprise Server CDs, which are numbered from CD 1 as well. In OES SP2, SLES 9 CD 1 is the same as OES CD 6.

5.
When the software is copied to your server, the installer asks for contact details of your LDAP server as shown in Figure 4.9. Enter these as needed and then click Next to proceed.

Figure 4.9. In the Local or Remote Directory Server screen, you need to provide contact details to contact a server holding a replica. These are needed to create the NCS-related objects in eDirectory.

[View full size image]

6.
Now select Existing Cluster to install the OES Linux server into an existing cluster, or New Cluster to create a new cluster. Enter the name of the cluster object as shown in Figure 4.10. Because the cluster already exists in eDirectory, you don't need to provide the other information. Only if you want to create a new cluster do you need to specify the cluster's IP address and, if you want to use a shared storage device (see Figure 4.11), the Linux device name of this device.

Figure 4.10. To add a node to an existing cluster, you just need to enter the eDirectory name of the cluster object.

[View full size image]

Figure 4.11. In the YaST Partitioner utility, you can see the device name of your iSCSI device.

[View full size image]

Tip

No idea what device name to use? From YaST, select System, Partitioner. This utility shows an overview of all storage devices known to your system. The iSCSI partition is displayed as the partition with the odd number as its type. See Chapter 7, "iSCSI," for more information about iSCSI.

7.
Now select the IP address of this node you want to use in the cluster. This should be the IP address with which the node is visible on the LAN and not the IP address assigned to one of the Fibre Channel interfaces. Make sure that the option Start Clustering Services Now is selected as well, and then click Next to continue. The necessary information is now written to eDirectory, and Novell Clustering Services is started on this node.

Adding a NetWare Node to a Linux Cluster

The description of how to add a NetWare node to a Linux cluster is easy: You can't. In a pure Linux cluster, no NetWare nodes can be added. Also in a mixed cluster, it is not possible to add any NetWare nodes.


Manually Starting and Stopping NCS

For management purposes, it can be useful to start and stop the cluster manually. On OES NetWare, the cluster is started with the LDNCS.NCF script, which is in SYS:\SYSTEM. To unload the cluster on OES NetWare, use ULDNCS.NCF. However, before you can successfully unload all cluster modules from a server, you have to issue the cluster leave command or the cluster down command.

A complete reference of available cluster commands can be found in Chapter 5, "Creating Cluster Resources."

On OES Linux, you can unload the cluster software with the cluster down command. To join the cluster again after you have brought it down, use the cluster join command.


Adding a License to Existing Cluster Nodes

If for any reason you have a problem with the licenses of one of the nodes in the cluster, it is possible to add a license later. Without a proper license, the node can't stay in the cluster and automatically leaves after a few minutes!

To add a license to an existing cluster node later, the following procedure can be used:

1. | Start iManager and log in as a user with administrative permissions.
2. | Select Licenses, Delete a License and browse to the location of existing NCS licenses. It is mandatory that these are removed from the tree before any new licenses are installed!
3. | After deleting existing licenses, select Licenses, Install a License. Now, on your workstation browse to the location of the license file.
4. | Select the license file and click Next. Then select the eDirectory container where you want to install the license file. Next, click Install to install the license in eDirectory and then click Done to complete the license installation.

Tip

Before buying, you can test to create your own two-node cluster. Licenses to create a two-node cluster are included with the evaluation version of Open Enterprise Server.


Removing Nodes from a Cluster

It may happen that you have to remove a node entirely from the cluster. The procedure for this is easy: Leave the cluster, unload all cluster-related modules, and remove all files that are activated when you start the cluster. The procedure for each of the OES flavors is described in the text that follows.

Removing an OES NetWare Node from the Cluster

When you're migrating a NetWare cluster to Linux, after a successful migration your NetWare node should be removed. Also, before taking out a NetWare node, you should remove it properly from the cluster. The following procedure describes how to perform this task:

1.
On the NetWare console, type cluster leave to leave the cluster.

2.
Enter ULDNCS to unload Novell Cluster Services.

3.
Delete the cluster node object from iManager (see Figure 4.12). This is the object added for this node in the cluster container.

Figure 4.12. One of the steps to remove a node from the cluster is to remove the cluster node object from the cluster container.

[View full size image]

4.
Remove the LDNCS line from AUTOEXEC.NCF.

5.
Disconnect the server from the shared storage.

6.
Reboot the server.

In this procedure the cluster-related NLMs are not removed from the server. If you want to remove them as well, make sure that the following modules are removed from SYS:\SYSTEM:

CLSTRLIB.NLMGIPC.NLM
SBD.NLM
SBDLIB.NLM
CRM.NLM
CMA.NLM
TRUSTMIG.NLM
CMON.NLM
LDNCS.NCF
ULDNCS.NCF
LDNCS.NCF
ULDNCS.NCF
VLL.NLM
TRUSTOOL.NLM

Removing an OES Linux Node from the Cluster

1.
On the OES Linux server, open a shell and enter the command rcnovell-ncs stop to stop the cluster daemons from running.

2.
Use the command insserv -r novell-ncs to remove the clustering services from all your runlevels.

3.
Start YaST, enter the root password if so required, and select Software, Install and Remove Software.

4.
From the drop-down list at the Filter option, select Selections, as shown in Figure 4.13. Next, from the list of software selections, select Novell Cluster Services. This displays the three components used by Novell Cluster Services on the right side of the screen.

Figure 4.13. On OES Linux, use YaST to remove the NCS software from your server.

[View full size image]

5.
Click twice on each of the software packages contained by Novell Cluster Services. As a result, you will see the symbol of a wastebasket in front of the name of the software package. Next, click Accept to start the removal of the software packages from your server.

6.
Reboot your server to complete the removal of this node.


Installing NCS in VMware

One of the system requirements of using Novell Cluster Services is that you must use some kind of common shared disk for the servers in the cluster. You can use shared SCSI disks, Fibre Channel, or iSCSI. That is all simple and easy if you have physical hardware. But how does that work in a virtual environment?

There can be many reasons why you would want to install NCS in a VMware environment. This can be for consolidation of your production servers, for setting up a test environment, or just for fun to learn how a cluster works and to study for your CNE exam.

What VMware Is

Let us first briefly explain what VMware technology is in case you have not used it before. With VMware products you create virtual machines on an existing operating system. These virtual machines are complete PCs. They have their own hard disk and networking capabilities. VMware does not emulate the host processor, but instead all threads are just handled by VMware on the host CPU. Therefore, the virtual machine uses the same CPU as the host. Other hardware is emulated in the form of an IDE or SCSI disk, an AMD network adapter, and a default VMware VGA adapter. Because the hardware is always identical, except for the processor, virtual machines are interchangeable between computers running VMware software.

VMware Usage Scenarios

The first possible clustering solution is to consolidate servers. Many companies do that nowadays with VMware's ESX Server product. This product comes with its own Linux-based operating system and does not rely on a default Windows or Linux host to run virtual machines. This isolates the virtual machine environment's stability from that of the host operating system.

Does it make sense to cluster-enable virtual machines that are already running in a very robust environment such as VMware ESX Server? We hear this question very often, and as all consultants we answer: It depends. With VMware ESX Server, and especially if you have two hardware boxes and the vMotion product, the availability you get is very high. With vMotion you can move a running virtual machine from one hardware box to another. But that does not happen automatically when a problem occurs. Also, it does not help with problems inside the guest operating system, Open Enterprise Server in this case. NCS can still help you to improve the availability of software components inside your virtual machine environment. If a server experiences a critical error, the other nodes in the cluster will take over the services. That can even be beneficial when all cluster nodes are being run on a single hardware box.

Clustering can also be performed with VMware GSX Server and VMware Workstation. These so-called "hosted" products run on top of a regular Windows or Linux server installation. Clustering is officially supported in VMware GSX Server, so that makes it a viable solution for running clustered Open Enterprise Servers inside virtual machines. For VMware Workstation, clustering is not supported, but you can still use that product to create a test environment or to set up a cluster for educational purposes.

VMware Clustering Requirements

Whatever product you have, there are two special configurations you need to take into consideration. First of all, you must use fixed-size disks. This is the only type of disk you can create in VMware ESX Server. For Workstation and GSX Server you can use a dynamically growing disk. But that would cause problems with your virtual machines. If they both write data into the virtual disk file and try to increase the size of the file, this would corrupt the data inside the disk. Therefore, you must use the setting Allocate All Disk Space Now when you create a new virtual disk to be shared between the virtual machines.

The other configuration to keep in mind is that you must configure the bus-sharing feature or the disk-locking feature. For the server products' clustering is supported, and they provide bus sharing (ESX Server) and SCSI Reservation (GSX Server). This is not supported for VMware Workstation; for that product you must disable disk locking and disk caching. These configurations are described later in this section.

VMware Clustering Configuration

VMware clustering configuration is detailed in the following subsections.

Adding a Shared Disk in VMware ESX Server

There are two possibilities to set up clustering for virtual machines in VMware ESX Server. You can either use a physical LUN (Logical Unit) from your SAN and assign that to two or more virtual machines, or create a new virtual disk file and add that to the virtual machines. Configuration of the virtual machines can be performed from the Management Interface, from the Remote Console utility, or from the VMware VirtualCenter management application.

Because the Management Interface will always be available, we decided to use that as an example in the following instructions. If you know how to work with the other two applications described before, the translation to those programs should not be too much of a problem.

The following instructions are based on a two-node cluster. The steps start with two already configured virtual machines. Instructions on how to set up virtual machines is available in the VMware documentation and in the VMware books that are readily available.

Perform the following steps to add a shared disk to the virtual machine:

1.
In the Management Interface, click on the virtual machine name (hyperlink) to open the configuration window for the first server. Go to the Hardware tab or select Configure Hardware from the pop-up menu to go to that tab directly.

2.
Click the link for Add Device at the bottom of the hardware configuration tab.

3.
Select Hard Disk as the device type to be added. This will start the Virtual Disk Configuration Wizard. In the wizard, select to create a blank virtual disk. You can also choose to use a physical LUN here. But for this example we use a shared virtual disk.

4.
In the next configuration window, select the location and the size of the virtual disk file. A minimum size of 20MB is sufficient for the SBD disk in Open Enterprise Server. Assign the disk to virtual SCSI adapter 1 (see Figure 4.14). Shared disks must be assigned to a separate controller. Leave the Disk Mode setting at the default setting of Persistent.

Figure 4.14. Create a shared disk and assign it to virtual SCSI adapter 1.

[View full size image]

5.
In the Hardware Configuration tab of the virtual machine, select Edit for SCSI Controller 1 that was added after adding the disk in the preceding step. Set the Bus Sharing option to Virtual (see Figure 4.15).

Figure 4.15. Set the Bus Sharing option to Virtual.

[View full size image]

6.
Open the Hardware Configuration tab for the second node in your cluster to add the shared disk to that node. Click the Add Device link at the bottom of the configuration screen. Add a hard disk to the machine, but now choose an existing disk and point to the shared disk that you created in step 4. As you can see in Figure 4.14, the disk must be assigned to SCSI node 1:0, just as was done for the other virtual machine.

7.
Now configure the SCSI Controller 1 that was added to the second node when you added the virtual disk to this controller. Set the Bus Sharing setting to Virtual, just as you did for the other node in step 5.

You have now configured the virtual machines to use the same shared disk. At this point you can install Novell Cluster Services in Open Enterprise Server.

Adding a Shared Disk in VMware GSX Server and VMware Workstation

The general setup for a cluster in GSX Server and Workstation is identical. There is only one special step you must take for each product, as described in the subsections that follow.

The instructions that follow are based on a two-node cluster. The steps start with two already-configured virtual machines. Instructions on how to set up virtual machines is available in the VMware documentation and in the VMware books that are readily available.

Follow these steps to add a shared disk to the virtual machines:

1.
For machine 1 open the virtual machine settings and click the Add button to add a new device.

2.
In the Add Hardware Wizard welcome screen, click Next.

3.
Select Hard Disk as the type of device and click Next.

4.
In the Select a Disk window, select Create a New Virtual Disk and click Next.

5.
Change the default setting to SCSI, and click Next. Whether the default is IDE or SCSI depends on the operating system that you have selected for the virtual machine. For NetWare it is IDE and for Linux it is SCSI.

6.
In the Specify Disk Capacity window, as displayed in Figure 4.16, select the proper disk size. A size of 500MB, noted as .5GB in the VMware selection box, is an adequate disk size for a small shared disk that can hold some data, and will be used as the disk for the Split Brain Detector. You can add additional fixed size disks to your cluster later, to store the data for your cluster application.

Figure 4.16. Select Allocate All Disk Space Now to create a fixed-size disk.

[View full size image]

Note

Do not forget to enable the option Allocate All Disk Space Now so that a 500MB file will be created on your hard disk.

7.
You will be warned that creating a preallocated disk may take a long time. If you are prepared to create the disk you have selected, click Yes.

8.
Specify a name for your disk, and select a location on your hard disk. After selecting the location of the disk file, click the Advanced button to specify the correct SCSI bus for your disk.

9.
In the Advanced Selection screen, select SCSI 1:0 for your SCSI disk, as displayed in Figure 4.17. We will use this extra SCSI bus, which will be added to your virtual machines.

Figure 4.17. Select SCSI 1:0 as the location on the virtual SCSI bus.

[View full size image]

10.
Click Finish. The fixed-size disk will now be created.

11.
Close the configuration control panel for the first virtual machine.

12.
Open the virtual machine configuration for your second server. Follow the same first three steps that you performed on your first server to add a new hard disk (steps 1, 2, and 3).

13.
In the Select a Disk window, select Use an Existing Virtual Disk.

14.
Browse the shared disk file (this is not the file with the -flat or -f00n suffix). Select the .vmdk file that you created in step 8. Also click the Advanced button to select SCSI1:0 as the location of your disk.

15.
In the Advanced Selection screen, select the SCSI 1:0 location for your SCSI disk.

16.
Click Finish and close the virtual machine settings control panel.

You have now configured the virtual machines with access to the same virtual disk. You must now follow the steps hereafter for either VMware GSX Server or VMware Workstation to enable clustering support.

Finishing The GSX Server Configuration

Clustering virtual machines is supported on VMware GSX Server by means of SCSI reservation. This must be enabled for the virtual SCSI bus where you have just added the shared disk. You must also disable the disk-locking feature to prevent virtual machines from locking a virtual disk, which would allow only one node to access the disk. Close the VMware GSX virtual machines to edit the configuration files manually.

Add the following lines to the .vmx files of both virtual machines:

disk.locking = "false"
scsi1.sharedBus = "virtual"

This setting enables SCSI sharing for the entire SCSI1 bus. If you do not want to share the entire bus, but instead you want to share individual disks on that bus, you can also configure an individual disk to be shared with the entry:

scsi1:1.shared = "true"

(1:1 specifies the second disk on the second SCSI bus.)

When you now power on the virtual machines, both servers will have access to the shared disk. This is the point where you can install Novell Cluster Services.

Finishing The VMware Workstation Configuration

Clustering virtual machines is not supported and there is also no support for SCSI reservation. You can, however, configure the virtual machines to start with the shared disk activated. And you must also disable the caching to prevent the virtual machines from corrupting the disk. Close the VMware Workstation virtual machines to edit the configuration files manually.

Add the following lines to the .vmx files of both virtual machines:

disk.locking = "false"
diskLib.dataCacheMaxSize = "0"

Both servers will have access to the shared disk when you power them on. This is the point where you can install Novell Cluster Services.


Chapter 5. Creating Cluster Resources

Now that the Novell Cluster Services (NCS) software has been installed on your server, it is time to move on and create some clustered resources. In this chapter you'll learn about the basics of cluster resource creation. Chapter 8, "Installing Cluster-Enabled Applications," provides in-depth information about clustering of some specific applications.


Creating Clustered Volumes

In many cases clustered resources do need clustered volumes. The reason for this is obvious: The clustered resource often needs to write data to a volume. If this is a local volume and the resource fails over to another server in the cluster because of a hardware failure, the resource wouldn't be capable of contacting its data. Therefore, a solution has to be provided to make the data available to the clustered resource, even if after a hardware failure the resource has failed over to another node in the cluster. The best way to handle this is by clustering volume objects. If you are working on an Open Enterprise Server (OES) NetWare cluster, these have to be Novell Storage Services (NSS) volumes. In an OES Linux environment, Reiser and ext3 volumes can be shared on the cluster as well. If, however, a NetWare cluster has to be migrated to a Linux cluster, only NSS volumes can be used on the Linux node. This is true for a mixed-node cluster as well.

Tip

Novell's NSS file system offers some advantages that other Linux file systems don't offer. For example, NSS allows you to work with the advanced Novell rights system, which isn't available on other Linux file systems. If the advanced NSS features are not needed in your environment, better use Reiser as the file system for your clustered volume. The advantage is that Reiser has less metadata to write when writing files and therefore is faster on file writes. Both Reiser and NSS are capable of handling large numbers of files in one directory. Ext3 is not as good a choice for handling large numbers of small files. Therefore, use NSS when the special NSS features are needed, and if they are not needed, use Reiser instead.

Creating Clustered Volumes on OES Linux and NetWare

After setting up the cluster, the first step to take is to set up the clustered volumes. This procedure involves three main steps:

1. | Create shared disk partitions
2. | Create NSS pools
3. | Create clustered volume objects

The following procedure describes all necessary steps. The procedure is applicable to NetWare as well as Linux. Any differences between the operating systems are marked as such.

Create Shared Disk Partitions

Before shared disk partitions can be created, Novell Cluster Services must be installed. After installation of Novell Cluster Services, you can set up the shared disk partitions with three different tools: iManager, Remote Manager, and the NSSMU tool. It doesn't matter which of these tools you use because they all lead to the same result. However, if you are configuring cluster volume objects on Linux, be aware that Remote Manager cannot be used.

In the following procedure you will read how to set up a shared disk partition with the NSSMU utility, which can be used from the OES NetWare as well as the OES Linux console. This procedure assumes that some kind of shared storage system already is available. If you don't have a system for shared storage in your environment, first read Chapter 7, "iSCSI," for more information on how to install an iSCSI shared storage system. If the shared storage system is available, apply the following steps to create a cluster volume object:

1.
On the server console of any server in the cluster, start the NSSMU utility by typing nssmu.

2.
Select the Devices option and press Enter to check whether the shared disk device that you need in order to set up NCS is already configured. In Figure 5.1 you can see what this looks like.

Figure 5.1. The Devices option in NSSMU helps you recognize the shared disk device.

[View full size image]

3.
On NetWare, return to the main menu of the NSSMU utility and select Partitions. In the overview of currently existing partitions, you will probably see a Cluster partition already. This is the partition used for the Split Brain Detector partition of your cluster. Click Insert to create a new partition. On Linux you do not need to do this. When configuring OES Linux, you can skip this step and create cluster pools immediately, as described later in this procedure.

4.
Now select the device on which you want to create the shared partition. The device name will be something like 0x12. After selecting the device, press Enter to continue.

5.
Next, select NSS as the partition type and press Enter. Then, as shown in Figure 5.2, enter the required size for the partition and optionally enter the NSS pool name as well. If you enter the pool name now, the NSS pool will automatically be created. In this procedure we will not create the NSS pool now; this will be done later. Now select Create and press Enter to create the partition that will be sharable for clustering.

Figure 5.2. When creating the shared partition, you can create the NSS pool immediately as well. Do, however, make sure that the state of the partition is set to Sharable for Clustering before continuing.

[View full size image]

Create Nss Pools

After creating the NSS partition that is marked as Sharable for Clustering, you can go on and create the NSS pool object. It is mandatory to create an NSS pool because only in an NSS pool can NSS volumes be created.

The following procedure explains how to create NSS pools:

1.
Start your browser and access iManager on https://yourserver/nps.

2.
Open Storage, Pools and make sure that the server on which you want to create the NSS pool is selected as shown in Figure 5.3. You will now see an overview of all pools that are available. Click New to create a new pool.

Figure 5.3. You can create an NSS pool directly in free space, or on NetWare in the NSS partition you've just created.

[View full size image]

3.
Now select the disk space where you want to create the pool. You can create a pool in an NSS partition, but it is also possible to create the pool directly in free disk space. The latter scenario will create the required partition automatically. Because we've just created an NSS partition and obviously want to use it, in this example the NSS partition is chosen. Be aware that on Linux the procedure to follow is slightly different: You should create the NSS pool directly on free disk space.

4.
Next you can specify the options to create the pool. Make sure that both Activate on Creation and Cluster Enable on Creation are selected. The latter option is especially important if you want to use the volumes in the pool in a clustered environment. After you click Next, some clustering parameters need to be entered, as shown in Figure 5.4. Make sure that the following parameters are given a value:

  • Virtual Server Name: All resources in the cluster environment need a virtual server name. This name is necessary for the end user because the end user wants to connect with the resource directly and not with the server that is currently hosting the resource in the clustered environment. The virtual server name for any clusterenabled NSS pool is automatically created, but you can change it to any name you like. Be sure to select a name that is unique on the network.

  • CIFS Server Name: This is the CIFS name that will be advertised to all CIFS clients on the network. This name is required only if CIFS is selected under the option Advertising Protocols.

  • IP Address: All cluster-enabled NSS pools need a unique IP address. This is the address on which the pool can be reached by end users, regardless of the server that is currently hosting the pool.

  • Advertising Protocols: You can select three protocols that allow end users to connect to the pool that is created in the cluster resource: NCP for native NetWare, CIFS for Windows clients and others that have the CIFS protocol stack installed, and AFP for Apple clients. Be sure to specify a CIFS server name as well if you want to enable the CIFS protocol as an advertising protocol. Support for all of these protocols is added to the cluster resource load script automatically.

Figure 5.4. While creating the NSS pool, you have to specify all important parameters that are needed for the pool as a cluster resource as well.

[View full size image]

5.
After specifying all required parameters, click on Create to create the NSS pool. Repeat this procedure for each additional pool that is needed.

Create Clustered Volume Objects

Now that the partition and pool are in place, you have to create one or more volumes in it. Before you later assign these volumes to other servers in the network, make sure that these servers have NSS installed; otherwise, you can't access the volume. By default, NSS is enabled on all OES NetWare nodes, but it is disabled by default on all OES Linux nodes.

We will now show you how to create NSS volumes from the NSSMU utility:

1.
From the NSSMU main menu, select the Volumes option. This will display an overview of all currently existing volumes, as shown in Figure 5.5.

Figure 5.5. After selecting the Volumes option from the NSSMU main menu, you see an overview of all volumes that are currently accessible from this server.

[View full size image]

2.
Press the Insert key on your keyboard and enter the name of the new volume. If you want to create an encrypted volume, answer Yes to the Encrypt Volume question that shows up. Otherwise, select No to continue. Now select the pool in which you want to create the volume and press Enter.

3.
You will now see an overview of all properties that can be set while creating the NSS volume. Make a selection of all options you need (see Figure 5.6).

Figure 5.6. On creation of a volume, you can set many options.

[View full size image]

Tip

You don't need to specify any of these options now. All options listed can be changed after creation of the volume.

A description of the available option is given here:

  • Backup: Select this option if you want to be able to use your backup software to create a backup of the volume.

  • Compression: Set this option to Yes if you want to be able to use compression on files or directories on the volume.

  • Data Shredding: Data shredding is the feature that overwrites deleted files with random bit patterns to make sure that the contents of the original file are no longer readable after deletion. Select Yes if you want to use this feature and next specify the number of times to shred data.

  • Directory Quotas: Select Yes if you want to be able to limit the amount of disk space occupied by a directory.

  • Modified File List: The modified file list allows backup software to make an incremental backup in a clever way. If your backup software supports this feature, be sure to enable this option.

  • Salvage Files: The salvage option allows administrators to undelete a file after it has been deleted accidentally. If you want to be able to use this very useful option, make sure that the default value Yes is selected here.

  • Snapshot - File Level: A file-level snapshot enables you to keep different versions of a file while you are working on it. The old versions of the file are stored in a MySQL database in that case. If you want to use this option, be sure to select it here.

  • User Space Restrictions: If you want to be able to limit the amount of disk space a user can write, be sure to select the User Space Restrictions option here. Notice that by default it is not enabled.

  • Flush Files Immediately: A file that is written to disk is written to cache first, before it is written to the hard drive of your server. This is a great feature for the write performance of your server; however, if a server crashes, all files still in cache that have not been written yet are lost. If you want to avoid losing files this way, be sure to enable the Flush Files Immediately option.

  • Migration: The migration feature allows file migration to a near-line storage media, such as a tape carousel. This feature is useful if your server has to deal with huge amounts of data that you don't want to store on the server's disk storage, but you do want to make sure that users can access this data without the intervention of a system administrator.

  • Read Ahead Count in Blocks: The read-ahead feature improves the read feature of a server. It allows a server not just to read the blocks that have been specifically requested, but to read the blocks directly after the requested block also. The reason for this is that, very often, the next blocks are needed as well. The default value for the read-ahead count is two blocks, which is a good value in most scenarios.

4.
After specifying which options you want to use, you still have to specify the size of the volume. You can set this at the Volume Quota option. By default, this value is set to 0, which allows the volume to grow to the pool size. If you don't want to use this feature and rather want to specify a fixed size, enter it here. Next, click Create to create the volume.

After creating the cluster-enabled pool, you can verify its existence from iManager. In iManager, select Clusters, Cluster Options and then browse to the cluster object. This will display a list of all objects that currently exist in the cluster. The NSS pool object is listed as one of them, as shown in Figure 5.7.

Figure 5.7. The cluster-enabled pool object is automatically added as a resource in the list of cluster objects.

[View full size image]

Choosing the Right File System Type for Cluster Volumes on Linux

On OES Linux, you can create clustered NSS volumes as described previously as well.

In a rolling cluster migration, it is even possible to mount an NSS volume created on OES NetWare on an OES Linux node. You can read the procedure given previously to learn all about the configuration of cluster-enabled NSS volumes on OES Linux; the NSSMU tool described previously is available in OES Linux as well. If, however, you don't need any of the specific benefits offered by NSS, you can cluster traditional Linux file systems as well. If you want to do so, it is recommended to use the EVMS volume manager on the shared disk device, instead of the LVM volume manager that is selected by default. The reason for this is that EVMS is created just for a clustered environment, whereas LVM is not. For that reason, you might be able to create clustered LVM volumes, but you might run into problems when using them in an operational environment because LVM isn't really supported. In the procedure given shortly, you will learn how to create an EVMS volume on your shared disk device and share it among the servers in your network. We will format this volume with the ext3 file system. In the next section, "Creating Clustered Resources," you will learn how to use these volumes in a clustered environment.

Using a volume manager in a clustered environment ensures that you can work on all nodes with the same volume names. It is, however, possible as well to create local device names for the clustered volume instead of working with the EVMS volume names. The disadvantage, though, of working with names like /dev/sdb is that it doesn't guarantee that the same device name is used on all nodes in the cluster. This poses a serious problem in cluster load scripts in which the same device name for the volume has to be referred to on all nodes in the cluster.

If you want to use a Linux file system in a clustered environment, you should think well before choosing the file system. Traditional file systems like ext3 and Reiser can be used in a clustered environment, but they cannot be accessed by more than one node at the same time. If you want to create a clustered volume that is accessible by more than one cluster node at the same time, you should use a file system that is designed for use in a clustered environment such as OCFSv2, GFS, or PolyServe. NCS offers support for OCFSv2.

The next procedure describes how to create an EVMS volume on one of the nodes in the cluster. It assumes that the shared storage device has already been set up properly. In this scenario, we describe how to create an EVMS volume that can be accessed by one node at a time; you will need to create a cluster resource and arrange in the load script of this resource that the EVMS volume will be mounted with the resource:

1.
From a node that you want to give access to the clustered volume, start YaST. Enter the root password when asked for.

2.
From the System menu in YaST, select the Partitioner utility. You will see a warning that mentions that you really have to be sure of what you are doing before you continue. Tell the software that you are capable by clicking Yes. The partitioner utility will now start.

3.
In the partition overview, you will see a list of currently existing devices. This list includes all local devices you can access, and shared storage devices as well. In Figure 5.8, device sda is the local storage device and sdb is an iSCSI partition created on an OES NetWare target (see Chapter 6, "Cluster Management," for more details). On this partition, there is an 11.9MB NetWare partition used for the Split Brain Detector process.

Figure 5.8. The YaST partitioner utility will show all local devices and shared devices on the network as well.

[View full size image]

4.
Now click the Create button and select the drive on which you want to create the clustered volume. Make sure that you select the shared device on the network and not a local device here, and then click OK to continue. Now indicate that you want to create a primary partition and click OK to continue.

5.
As in Figure 5.9, you will now see the screen in which you can enter all properties for the EVMS partition you want to create. Use the following specifications and then click OK to continue:

  • Select Do Not Format.

  • In the File System ID drop-down list, choose 0x8E Linux LVM.

  • In the Size area, select End and specify the size of the partition you want to create. For example, if you want to create a 30GB partition, enter the value +30GB.

Figure 5.9. If you want to create an EVMS volume, start by creating an LVM partition.

[View full size image]

6.
You have now created a partition that is ready for EVMS. Now let's go on and create some EVMS volumes on this partition. This is a two-step procedure. First you have to create an EVMS container. The EVMS container can be used to gather available storage from different devices. We won't do that here, but in all cases, an EVMS container is needed anyway. After creating the EVMS container, you can create the EVMS volumes in it. To create an EVMS container, make sure that in the Expert Partitioner screen, the Linux LVM partition you've just created is selected, as shown in Figure 5.10. Then click the EVMS button.

Figure 5.10. Make sure that the LVM partition is selected and then click EVMS.

[View full size image]

7.
Now the EVMS configurator is started. On this screen, select Create Container. As shown in Figure 5.11, this opens a new screen where you can enter the properties of the EVMS container. An important property is the physical extent size. You can best compare this to the block size that you enter when creating a logical volume. By default, the physical extent size is 4MB. This size determines that the size of a logical volume always needs to be an increment of 4MB. A 4MB physical extent size allows you to create a logical volume with a maximum size of 256GB. You need more than that? Then use a bigger physical extent size. Use, for example, a 16MB physical extent size to create volumes of 1TB. The maximum volumes size is always 65,523 times the physical extent size.

Figure 5.11. From the Create EVMS Container screen, you have to enter all properties for the container you want to create.

[View full size image]

8.
From the Device list, select the LVM partition that was created on the shared storage device. In many cases, /dev/sda will be used for the local device, so the name of the device will be sdb, sdc, or higher, followed by a number. In Figure 5.11, we need to select the sdb2 device. Now enter the container name (system is a good choice) and click Add Volume. As you can see in Figure 5.12, the EVMS device name is added to the LVM partition you created earlier. Now click OK to close the screen. This will bring you back to the EVMS configurator main screen.

Figure 5.12. After you've added the EVMS container to the LVM partition, the container name will be added in the device overview.

[View full size image]

9.
Now you have to create some logical volumes in the EVMS container. These logical volumes will be the storage devices in which the data is stored. Make sure that in the EVMS Container drop-down list, your EVMS container is selected, and then click Add to add a new EVMS volume. As shown in Figure 5.13, this will display the Create Logical Volume screen, where you have to enter all the properties you want to use for the new partition.

Figure 5.13. In the Create Logical Volume screen, you can enter all required properties for the logical volume you want to create.

[View full size image]

10.
Enter a value for the following properties and then click OK to continue:

  • Volume Name: This will be the device name of the volume. You can find this name under /dev/evms/lvm/evms-container-name in the /dev directory on the local file system. For example, if the EVMS container has the name system, and the name of the volume is data, the full EVMS volume device name will be /dev/evms/lvm/system/data.

  • Size: Enter the required size of your volume here.

  • Stripes: The number of stripes determines the number of devices the data is written to. If you have just one device in the EVMS container where the volume is created, it doesn't make sense to select anything other than 1. If, however, the EVMS container consists of four devices, it makes sense to select 4; this will force data to be written to all of these four devices at the same time and increase the speed of the volume.

  • Stripe Size: If you are working with just one stripe, the default stripe size is 64KB. If, however, you are working with more than one stripe, you can select the stripe size. The minimal stripe size is 4KB, the maximum is 512KB. If speed is an issue when working with many small files, select a small stripe size; if storage is an issue when working with huge files, select a bigger stripe size.

  • Fstab Options: Here you can specify options for the file system when it is mounted from fstab. Because the logical volume will not be mounted from fstab, this option can be ignored.

  • Mount Point: This option is used by fstab to specify on which directory the volume needs to be mounted from fstab. The software needs to know the name of a directory here. We recommend that you use a dummy directory because later you will remove this line from fstab anyway.

  • Format: Here you indicate what file system you want to use on the clustered volume. You can use ext3, reiser, xfs, or jfs. Because it is a widely supported file system, we recommend that you ext3; if, however, you have a special reason to choose another file system, you can select it.

  • Encrypt File System: Some file systems can be encrypted. For speed issues, we recommend that you not use encryption on a clustered file system.

11.
If required, repeat the procedure to create more clustered volumes. When finished, click Next to continue. Then from the Expert Partitioner main screen, click Apply to apply your settings. In the screen with the overview of changes that will be applied, click Finish to apply your changes. The modifications will now be written to the shared storage device.

You now have created an EVMS volume that can be used in a clustered environment. There are, however, still some tasks that need to be performed before Novell Cluster Services can use this volume on all nodes in the cluster. This is because the YaST installation procedure for EVMS volumes assumes that the volume is on a local disk. Therefore, YaST makes accessing the volume easier by putting all information required to access the volume in the configuration file /etc/fstab. This ensures that the volume is mounted automatically when the server is started. If the volume is on a shared disk device in a clustered environment, it must not be mounted from YaST automatically, but from the scripts that load the cluster resources that use these volumes. The reason for this is obvious: You don't want your local machine to manage these volumes, but the cluster resource. Also, you have to make sure that the EVMS service is activated on booting of all nodes in the cluster that need to access the volume. The next steps explain how to finish the installation of the EVMS volumes:

1.
On the server where you've configured the EVMS volume, start an editor and open the file /etc/fstab in it. In this file, you see a line that takes care that the clustered volume is started automatically. This line looks like the following example. Comment this line out so that the volume will not be activated from fstab when the system boots. Also write down the device name of the EVMS volume somewhere; you'll need it later:

/dev/evms/lvm/system/data    /somwhere    ext3    acl,user_xattr      1 2

2.
Now you need to make sure that the EVMS service is started on all machines that need access to the shared volume. On all of these machines, from YaST, System, start the Runlevel Editor. In the Runlevel Editor, select Expert Mode and locate the boot.evms service, as shown in Figure 5.14. Make sure that this service is started when the system boots, by selecting the option B (you can use the Alt+B key sequence to access this option). Then click Finish and Yes to apply the changes.

Figure 5.14. You need to make sure that the boot.evms script is activated on boot of all machines that need access to the shared storage.

[View full size image]

3.
Issue /etc/init.d/boot.evms on all machines to start the EVMS service and make sure that all can access the shared volume.

4.
After starting the EVMS service, make sure that the clustered volume can be accessed from other machines as well. First open a command line, and then use the command mount /dev/lvm/evms/system/data /mnt to verify that the volume can be activated on all machines. Be sure to enter the volume name that is used on your machine when issuing this command. If the volume can be mounted, you can go on and use it with clustered resources on OES Linux.


Creating Cluster Resources

Now that you have created cluster volumes, you can continue and configure resources that use these volumes. We will now discuss how these resources can be configured on OES Linux and OES NetWare.

Clustering a Simple Application on OES NetWare

You will now learn how to install a cluster resource on OES NetWare. We have chosen not to cluster a very complex application, but an easy one: the Dynamic Host Configuration Protocol (DHCP) server. Even if you don't plan to cluster the DHCP server that runs on OES NetWare, we recommend that you read this section anyway; some useful information is provided about creating clustered resources.

This section assumes that the DHCP server is already installed on one of the servers in your network. If it is not, install it before you continue with the following procedure. Note that the DHCP server is kind of a special cluster resource because it doesn't need a shared storage device. This is because the DHCP server stores all its information in eDirectory. Read Chapter 8 for more information about cluster services that do need shared storage devices. Apply the next procedure to create the DHCP server as a cluster resource on NetWare:

1.
After installation, the DHCP server is started by the server on which it runs. This has to change because NCS now has to manage the DHCP server. Therefore, make sure to stop DHCPSRVR.NLM from the console of your server and to prevent it from starting automatically from autoexec.ncf as well.

2.
From a workstation, start iManager and access your server. Enter the name and password of an administrator account to log in to iManager. Then from iManager Roles and Tasks (left part of the screen), select Clusters, Cluster Options.

3.
From the Cluster Options window, as shown in Figure 5.15, use the magnifying glass to select your cluster. The cluster is added to the eDirectory container you specified on installation with its own eDirectory object.

Figure 5.15. From the Cluster Options window, select your cluster object.

[View full size image]

4.
Next, you have to add a cluster resource object for the DHCP server you want to cluster. From the list of cluster objects, click New. In the window that opens (see Figure 5.16), you can select from three types of cluster objects. Select the Resource object and then click Next to continue. Following is a short list of available objects:

  • Pool: Use this to create an NSS pool as a clustered object. Note that you don't need this interface to create an NSS pool as a clustered object; as discussed earlier in this chapter, you can add NSS pools as clustered objects from the NSSMU interface as well.

  • Resource: Use this option for any resource you want to cluster. This option is used for almost all services that have to be managed by the cluster.

  • Template: To make creation of clustered resources easier, some template objects are provided by default. In these template objects, default settings for some services are provided. An administrator can add templates as well.

Figure 5.16. Before creating a clustered resource, you have to pick the type of resource you want to create.

[View full size image]

5.
Now specify a name for the resource. You can also select a template for the cluster resource object to inherit some default settings from. Use the magnifying glass to select a template object from the cluster container object. The following templates are available by default:

  • DHCP Server: Use this template to cluster the NetWare DHCP server.

  • Generic IP Service: Use this template for all services that don't have a default template.

  • iFolder Server: Use this template to cluster the iFolder 2 server on NetWare.

  • MySQL Database Server: Use this template to cluster the MySQL database server on NetWare.

Note

The list of available templates may change in future releases.

6.
Select the DHCP Server template as shown in Figure 5.17. Next, make sure that the Define Additional Properties option is selected. If you want the cluster resource to come online after creation, select the corresponding option as well. It is also possible to bring the cluster resource online later from the Cluster Manager window. Click Next.

Figure 5.17. When you're creating a cluster resource, it is recommended that you select a template to inherit settings from.

[View full size image]

7.
Now the cluster load script is displayed, as shown in Figure 5.18. This is a default cluster load script that can be used to activate a generic DHCP server. From this load script, the cluster commands are called that are needed to activate the DHCP server.

Figure 5.18. In the cluster load script, you specify how a resource has to be loaded.

[View full size image]

8.
The cluster load script to activate the DHCP server is not complicated. There are mainly three lines that are used to start the server:

CLUSTER DHCP CN=SERVER.O=ORG.T=TREE
add secondary ipaddress A.B.C.D
dhcpsrvr -d3 --servaddr=A.B.C.D

9.
On the first line, the CLUSTER DHCP command is used to pass important information about the DHCP server to another server in the cluster when the service fails over. Edit this command to add the complete distinguished name of the server in your tree that has DHCP configured. If, for example, your server is cn=oes-netware1.ou=oesnetware1. o=oes, and the server is in OES-TREE, this command should read CLUSTER DHCP CN=OES-NETWARE1.OU=OES-NETWARE1.O=OES.T=OES-TREE

10.
Edit the add secondary ipaddress command to assign an IP address that can be used by the DHCP server: Every clustered resource on your network needs an IP address of its own. This is the IP address the end user communicates with. Make sure that it is a unique address in the same address range as the address of your server.

11.
Edit the command dhcpsrvr -d3 --servaddr=A.B.C.D to work with this same IP address as well.

Tip

The default parameter -d3 will enable the highest logging level and will log all information to a file also. For daily operation it is better to set this parameter to -d1 to show only basic information to the server screen.

12.
Next enter the Timeout value. This is the duration of time the cluster software tries to load the resource before generating an error message. Then click Next to continue.

13.
Now you see the unload script as displayed in Figure 5.19. This script is used to make sure that the service is unloaded properly when it has to migrate from one node to another. In this sample script, just one line needs to be edited, and that's the line where the secondary IP address is deleted. Make sure that it reflects the IP address you've assigned in the cluster load script. Then click Next to continue.

Figure 5.19. All clustered resources need an unload script to unload properly as well.

[View full size image]

14.
Next, as shown in Figure 5.20, you need to specify the resource policies. Make a selection and then click Next to continue. The following options are available:

  • Resource Follows Master: This resource will always run on the cluster master node.

  • Ignore Quorum: Normally, for a service to come up in the cluster, a minimal number of cluster nodes has to be available. This minimum is defined as the quorum. If you want to overwrite the quorum for this specific resource, select the Ignore Quorum option.

  • Start Mode: Select Auto to start this resource automatically when the cluster comes up, or Manual if you want to start the resource manually.

  • Failover Mode: If this option is set to Auto, the resource is automatically migrated to another node in the cluster if the node it is currently running on goes down. If you set this option to Manual, you have to migrate the resource manually to another node after a failure.

  • Failback Mode: The default option of Disable indicates that nothing should happen if the original node the resource ran on before that node failed comes back. Select Manual if you want the administrator to decide manually what happens and Auto if you want the resource to automatically return to the node it was originally running on.

Figure 5.20. The Resource Policies screen specifies how the resource is used in the cluster.

[View full size image]

15.
In the final step, you have to assign nodes to the clustered resource, which you can see in Figure 5.21. By default, all nodes in the cluster will be assigned. Use the arrow keys to remove nodes from the list that you don't want the resource to be activated on. Failover will take place in the order in which the nodes are assigned: The service will run on the first node, and in case of problems it will fail over to the second node, and so on. Use the arrow keys to change the order. Make sure that in a mixed environment only the operating system that the application is written for is selected; it doesn't make sense to try to run the NetWare DHCP server on a Linux server. Click Finish when finished. This will add the newly created service to the list of cluster objects.

Figure 5.21. By default, all nodes in the cluster are assigned to newly created resources.

[View full size image]

You have just created a cluster resource. Depending on the settings you have selected, it may or may not start automatically. You can verify this from the Cluster Manager, which you can select from iManager Roles and Tasks, Clusters, Cluster Manager. In the Cluster Manager screen, you can see an overview of all nodes and resources in the cluster and their current state. An example of this overview is shown in Figure 5.22.

Figure 5.22. In Cluster Manager you can see an overview of all nodes and resources in the cluster.

[View full size image]

Clustering a Simple Application on OES Linux

If you know how to cluster a resource that runs on NetWare, you can cluster a resource to run on Linux as well. Many options are just the same, including the syntax needed to load and unload services. Not the same is that on OES Linux, traditional Linux file systems such as ext3 and Reiser can be used for shared storage. In the next procedure you can read how to cluster the DHCP server on OES Linux.

The DHCP server on OES Linux is not the same as the DHCP server on OES NetWare. Whereas the DHCP server on NetWare writes all data to eDirectory, on OES Linux all data is written to some files. These files are stored on the shared volume we created earlier in this chapter. The DHCP server itself must be installed on both servers. On each of these servers, a mount point must be created to refer to the shared storage device. The following procedure describes how this works (before starting this procedure, make sure that the DHCP server software is installed on both servers):

1.
On the shared disk, create a volume that can be dedicated to the DHCP server's data. Refer to the section "Creating Clustered Volumes on OES Linux and NetWare," earlier in this chapter, for information on how to create this shared volume.

2.
On both servers, create a mount point where the shared volume can be mounted. In this example, we assume that a folder with the name /media/dhcp exists for this purpose.

Tip

OES Linux would expect these mount points to be in /media. If NSS is enabled on OES Linux, it would be in /media/nss.

3.
Log on as root and mount the shared volume. Depending on the name of the volume, you could use the following command:

mount /dev/evms/lvm/system/dhcp /media/dhcp

4.
Next, you need to create the data structure where the DHCP server writes its data. On the directory where you've just mounted the shared volume for the DHCP server, use the following commands:

mkidr etc
mkdir -p var/lib/dhcp

5.
Use the YaST option Network Services, DHCP Server to configure the DHCP server. From a command line, use the rcdhcpd start command to verify that the DHCP server can start without errors. After verifying this, use the rcdhcpd stop command to stop the DHCP server.

6.
The configuration files of the DHCP server are still stored in the /etc directory. For use in the clustered environment, you need to make sure that these are copied to the shared volume. Do this with the following command:

cp /etc/dhcpd.conf /mediadhcp/etc

7.
The dhcpd.conf file is not the only file the DHCP server needs. It needs the dhcpd.leases file as well. This file is used to keep track of current usage of IP addresses that are issued by this DHCP server. In a clustered environment, this file needs to be on the shared storage as well. You don't need to copy an existing file; instead you have to create a new and empty file. Use the following command to create this file:

touch /mediadhcp/var/lib/dhcp/dhcpd.leases

8.
After doing all necessary preparation, you can create the cluster resource for the DHCP server. Before starting, you have to make sure that the shared volume is unmounted. Do this by using the following command:

umount /mediadhcp

9.
Now open your browser, go to the iManager URL on https://yourserver/nps, and log in as a user with administrative permissions. Then select Roles and Tasks, Clusters, Cluster Options. Select the cluster you want to create the resource for.

Tip

If you have selected this cluster before from iManager, use the History button for easy access to data entered previously. The History button is the button to the right of the magnifying glass next to the option where you select the cluster.

10.
Click New and then select the resource type to create the cluster resource. Click Next to continue.

11.
Enter a name to assign to the cluster resource and then select the template to inherit settings from. OES Linux is provided with a template to create a DHCP server resource to make life easier for the administrator. Select this template, make sure that the Define Additional Properties box is selected, and then click Next to continue.

Note

In a mixed-node environment, you can find that only NetWare templates are available. In that case, you cannot select the template object and you must enter all parameters yourself.

12.
Now you see the cluster load script screen. Here you have to enter all commands that make sure that the cluster resource is loaded properly. The following script gives a good example of what should happen:

#!/bin/bash
. /opt/novell/ncs/lib/ncsfuncs
exit_on_error mount -t ext3 /dev/evms/lvm/system/dhcp /mnt/dhcp
exit_on_error add_secondary_ipaddress 192.168.0.153
exit_on_error /usr/sbin/dhcpd -cf /mnt/dhcp/etc/dhcpd.conf -lf\
       /mnt/dhcp/var/lib/dhcp/dhcpd.leases
exit 0

This script may appear complex but it really is not. As the first line indicates, this script is in bash syntax (see also Figure 5.23). On the second line, NCS-specific functions are called. These functions are all commands that can be used in an NCS load script on a NetWare environment. This allows an administrator to use the same commands both on NetWare and on Linux. On the third line, the shared volume is mounted. The mounting of this volume starts with the command exit_on_error, which comes from the ncsfuncs functions script. The exit_on_error statement makes sure that the script stops immediately if there is an error. Next, an IP address is added that can be used by the DHCP server. Then the DHCP server is activated. Notice the options -cf (configuration file) and -lf (leases file), which are used to refer to the two configuration files that are stored in a location different from where they are normally. Finally, the exit 0 command is used to tell the shell that the script has run successfully. Modify your load script as neededand don't forget to set the timeout value to a few minutesand then click Next to continue.

Figure 5.23. On OES Linux, the cluster load script is a bash script.

[View full size image]

13.
Now you have to create the unload script as well, as shown in Figure 5.24. To stop the DHCP server started with the foregoing load script, the following unload script can be used:

#!/bin/bash
. /opt/novell/ncs/lib/ncsfuncs
ignore_error killproc -p /var/run/dhcpd.pid -TERM /usr/sbin/dhcpd
sleep 8
ignore_error fuser -k /mnt/dhcp
ignore_error del_secondary_ipaddress 192.168.0.153
exit_on_error umount /mnt/dhcp
exit 0

Figure 5.24. In the unload script, NCS makes sure that the resource is stopped, no matter how.

[View full size image]

This unload script makes sure that the DHCP resource is stopped, no matter how. Notice the ignore_error statements that are used in the script to make sure that execution of the script is not interrupted and the script keeps trying. On the third line of the script, the killproc command is used to make sure that the DHCP server is terminated. If this does not work within 8 seconds, another method is used to terminate the services anyway, this time by using the Linux command fuser. This command makes sure that all processes that are currently using files in the /mnt/dhcp directory are terminated. Next, the secondary IP address belonging to the DHCP server is deleted. Then with an exit_on_error statement the umount of the shared volume is accomplished. There is a good reason an exit_on_error statement is used to accomplish this task: If the volume could not be deactivated from this server, it is impossible for the other server to mount it and the service will not be able to load on the other server. The exit_on_error makes sure that the script terminates and starts all over again if the shared volume could not be unmounted.

14.
In the last step, the resource policies have to be set. The following options are available:

  • Resource Follows Master: This resource will always run on the cluster master node.

  • Ignore Quorum: Normally, for a service to come up in the cluster, a minimal number of cluster nodes has to be available. This minimum is defined as the quorum. If you want to overwrite the quorum for this specific resource, select the Ignore Quorum option.

  • Start Mode: Select Auto to start this resource automatically when the cluster comes up, or Manual if you want to start the resource manually.

  • Failover Mode: If this option is set to Auto, the resource is automatically migrated to another node in the cluster if the node it is currently running on goes down. If you set this option to Manual, you have to migrate the resource manually to another node after a failure.

  • Failback Mode: The default option of Disable indicates that nothing should happen if the original node the resource ran on before that node failed comes back. Select Manual if you want the administrator to decide manually what happens and Auto if you want the resource to automatically return to the node it was originally running on.

15.
In the final step, you have to assign nodes to the clustered resource. By default, all nodes in the cluster will be assigned. Use the arrow keys to remove nodes from the list that you don't want the resource to be activated on. Failover will take place in the order in which the nodes are assigned: The service will run on the first node, and in case of problems it will fail over to the second node, and so on. Use the arrow keys to change the order. Make sure that in a mixed environment only the operating system that the application is written for is selected; it doesn't make sense to try to run the Linux DHCP server on a NetWare server. Click Finish when finished. This will add the newly created service to the list of cluster objects as shown in Figure 5.25.

Figure 5.25. After creation, the new cluster resource will be added to the list of available resources.

[View full size image]

Now that you have successfully created the cluster resource, it can be started from the Cluster Manager screen. Select the resource check box there and click Online in the menu above to bring it online. This should change the state of the resource from offline to running. In case you have any problems, verify the following:

  • Check that the DHCP server currently is not running on either node.

  • Check that the shared volume is not currently mounted anywhere.

  • Verify that you haven't made any errors in the cluster load script.

  • Check /var/log/messages for any error messages that may indicate why the script does not load (see Figure 5.26).

Figure 5.26. If the resource does not load properly, check /var/log/messages for clues.

[View full size image]


Cluster Script Syntax

An essential part of clustering is the cluster load and unload scripts. From the moment that NCS is available on OES NetWare as well as OES Linux, the syntax of the cluster script is bash, and the default shell that is available on both NetWare as Linux. On NetWare some specific NCS-related commands were already available; on Linux these commands are added to a file that is called from each cluster load or unload script. This is the line:

. /opt/novell/ncs/lib/ncsfuncs

In this script the following functions are defined:

  • del_secondary_ipaddress: This command can be called from the cluster load script to remove a secondary IP address that was added for a specific service.

  • add_secondary_ipaddress: This command is used to add a secondary IP address. This IP address is needed because all resources that run on the cluster need an IP address that is independent from the IP addresses already in use on the servers in the network.

  • exit_on_error: This command makes sure that the script is stopped if an error occurs. If the timeout value assigned to the cluster resource has not been surpassed already, the script will run again until it succeeds or the timeout value is passed.

  • ignore_error: In some cases, errors must be ignored. This feature is often used in cluster stop scripts. The reason this statement is used is that in a cluster stop script it is important that the resource be stopped, and it is not really important to do it in the best possible way.

In the resource load and unload scripts, some cluster-specific commands can be used as well. These are the commands you can use from the console as well to manage the cluster and its resources. An overview of these commands is provided in the next chapter.


Summary

In this chapter you learned how to create some cluster resources that are not too complex. You read how to set up shared volumes on NetWare and on Linux as well. Also you learned how to cluster the DHCP server on both platforms. At the end of the chapter, a summary was given of the functions that can be used in resource load and unload scripts. In the next chapter, you will get more details on how to work with the cluster management applications.


Chapter 6. Cluster Management

In the past, management of cluster resources was available from many utilities. Remote Manager, iManager, and ConsoleOne could be used. For your NetWare server you can still use ConsoleOne and Remote Manager; in Open Enterprise Server (OES) for Linux, iManager is the main management utility for the cluster environment. There are also some options to manage Novell Cluster Services (NCS) from the command line. These options are dealt with in this chapter. Also, logging, event history, and alerting are covered.


Cluster Resources Management Applications

In the old days, an NCS cluster was managed from ConsoleOne. Nowadays, there are no snap-ins available to manage NCS from ConsoleOne. Management of cluster resources from Remote Manager is supported only on OES NetWare servers. Therefore, the main focus of this section is on management of cluster resources from iManager.

Managing Cluster Resources with iManager

In an Open Enterprise Server environment, iManager is the main management utility for all eDirectory-related resources. Therefore, iManager is the tool to use to manage cluster resources as well. You can access iManager from a supported browser like Mozilla Firefox or Internet Explorer. The URL to access iManager is https://yourserver/nps. iManager does not have to run on a cluster node itself; it can also be run from other OES servers in your network, as long as they are in the same tree.

Tip

iManager doesn't work properly with Mozilla because it ships with the unpatched version of OES Linux. You must update iManager before it can be used on the server itself. However, if you have a workstation, you can download the latest version of Mozilla and use that instead.

After entering the required URL, you have to specify the following information:

  • Username: This is the eDirectory name of the account you want to log in with to manage your cluster environment. Complete distinguished names such as .admin.oes are not supported; use a relative distinguished name like admin.oes instead.

  • Password: This is the password of the given user.

  • Tree: Here you can enter the name of the tree you want to log in to. This will work, however, only if Service Location Protocol (SLP) is active in your network. Instead of entering the name of the tree, you can enter the DNS name or the IP address of the server you want to log in to (see Figure 6.1).

    Figure 6.1. If you have problems connecting to a tree with SLP, use the IP address or DNS name of a server that contains a replica instead.

    [View full size image]

After logging in to iManager, you see an overview of all tasks available to your account in the bar on the left side of the screen. This bar is titled Roles and Tasks. Under the Clusters heading in this sidebar, three management categories are available:

  • Cluster Manager: From this screen you can manage resources currently configured in the cluster. If, for example, you need to migrate a cluster resource, you will do it from this screen.

  • Cluster Event Log: On an active cluster lots of events can occur. From the Cluster Event Log you can find an overview of all recent events.

  • Cluster Options: This is the screen where generic settings for the cluster are configured. Use it to specify cluster parameters or create new cluster objects.

Administering Cluster Resources and Generic Parameters from the Cluster Options Interface

In Chapter 5, "Creating Cluster Resources," you read how to create a new cluster resource from the Cluster Options screen. From this screen, all properties of the cluster itself and the resource in the cluster can be edited as well. Before you can do any of these, you have to select the cluster object in eDirectory. After selecting this object, you will see an overview of all objects such as cluster nodes and cluster resources that exist in this container. Then you can modify the properties of all of these objects.

Tip

For those new to iManager, here's a short description of everything you need to know to select objects as shown in Figure 6.2. In the Select Object screen, you will always see a magnifying glass. This is used to select objects in eDirectory. If you click it, a browser is started. In this browser you typically see objects hyperlinked and container objects preceded by an arrow. If you need to browse to another container, use the arrow keys for browsing. If you want to select an object, click on the hyperlink.

Figure 6.2. Make sure that the cluster object is selected before modifying any of its properties.

[View full size image]

The following procedure describes how to modify properties of the cluster object itself:

1.
Make sure that the cluster resource is selected, as shown in Figure 6.2.

2.
After selecting the cluster object, click the Properties button to access the cluster properties. This will show all generic properties for your cluster object on four different tabs.

3.
On the General tab, three kinds of properties are available, as shown in Figure 6.3:

Figure 6.3. On the General tab, generic settings such as the quorum triggers are applied.

[View full size image]

  • First, you can enter the unique IP address of the cluster and its port address. Change these when the IP address and port that you originally used must be changed.

  • The second property is about the quorum. This is an important property: It specifies the number of nodes that have to be present in the cluster before a resource starts loading. By default, it will wait for all nodes in the cluster to come online before any resource is loaded, but it will do this only for the period specified as the timeout period. Imagine, for example, that the number of nodes defined in the quorum is set to four, and the timeout limit is set to one minute. If after one minute only two nodes are present in the cluster, resources will start loading on these two nodes anyhow. Often the default quorum of all nodes in the cluster works fine. You should always make sure that at least two nodes are used in the quorum. If only one node is specified, it implies that all cluster resources are activated on the first node that comes up in the cluster.

  • The third parameter you find on the General tab is about notification. Here you can choose the option Enable Cluster Notification Events. By default, this feature is disabled. If you choose to enable it, you can use the plus sign next to the Email Addresses drop-down list to add new email addresses. All addresses specified here will receive automatically generated alerts when critical events are happening in the cluster. By default, messages are sent only if a critical event occurs. It is, however, also possible to generate verbose messages. By default, the messages are sent in plain-text format. You can also indicate that the XML format is used for the automatically generated messages.

4.
Next, activate the Priorities tab. Here you will find a list of all resources currently configured in the cluster (see Figure 6.4). This list determines which resource is loaded first. Make sure that your most important resources are at the top of this list and resources that are less important are lower in the list. Also think of resource dependencies: If a given resource relies on another resource to be loaded first, you can specify that here.

Figure 6.4. Make sure that the most important resource in your cluster is listed first on this tab.

[View full size image]

5.
On the Protocols tab shown in Figure 6.5, important protocol settings can be specified. Use these settings to indicate how nodes in the cluster should communicate. These settings are important because among other settings, they determine what happens if the master node in the cluster goes down. After changing any of these settings, make sure that the Cluster Services software on all nodes is restarted to activate the new settings.

Figure 6.5. The protocol settings determine how nodes in the cluster will communicate.

[View full size image]

The first parameter found here is the Heartbeat parameter (heartbeat rate_usecs in the configuration script). This setting specifies how often nodes that are not the master node will send a signal that they are alive to the master node. The Tolerance rate (censustaker tolerance) is related to that: It specifies the amount of time the master node gives other nodes in the cluster to indicate that they are still alive. If the master node doesn't hear from any of the other nodes within this period, it will automatically remove the other node from the cluster. Next, the Master Watchdog (sequencer master_watchdog) parameter specifies how often the master node should transmit a signal on the network to indicate that it is still alive. The default setting is set to one second, which indicates that the master will tell all other nodes every second that it is still there. The Slave Watchdog (sequencer slave_watchdog) parameter is related to this: It indicates the amount of time the master node has to indicate that it is still there. If it fails to do so, the master node is removed from the cluster and one of the other nodes will become the master. The Maximum Retransmits (sequencer retrans_max) option is currently not used in NCS, but probably will be used in the future.

At the bottom half of the Protocols screen, you will see the script that is generated automatically when any of these parameters is changed. Some of these options have not been discussed yet:

  • panning clusterid: This is a unique identification number for the cluster. It is the same on all nodes that are in a cluster, and it should be different on nodes that are members of other clusters. For all nodes, the panning clusterid is normally set automatically when the node comes up and joins the cluster. If a node ever refuses to join the cluster, you need to restart the node, which will automatically correct the problem.

  • nodeid: For all nodes in the cluster, there is a nodeid line. It indicates the ID of the node, as well as its priority in the cluster.

6.
Last, there is the Business Continuity tab. This tab relates to the Business Continuity product, which is covered in Chapter 12, "Introduction to Business Continuity Clustering."

7.
To save all modifications, click OK.

The next option you will encounter in the Cluster Options screen is the Repair button. This option is needed when there are problems related to the cluster objects: It will automatically repair all related objects in the eDirectory tree, thus making it unnecessary to run an ndsrepair session manually.

Last, there are the options specific to the different cluster resources. To access options of any of these objects, select the check box in front of the resource object and then click the Properties button. It will display all properties related to the object. The options shown are the same as the options specified during creation of cluster resources; consult Chapter 5 for more details about them.

Managing Cluster Resources From the Cluster Manager Screen

The goal of the Cluster Manager screen, shown in Figure 6.6, is to get an overview of the current state of the cluster. From this screen, you will immediately see the status of all nodes in the cluster and the related resources. The screen is basically all you need to see whether the cluster is running smoothly. For your convenience, the screen is automatically refreshed as well.

Figure 6.6. From the Cluster Manager, an overview of the current state of the cluster is provided.

[View full size image]

On top of the screen, you will find the Run Report button. After selecting the cluster you want to monitor, you can use this button to get an overview of the current state of the cluster, as displayed in Figure 6.7. The cluster report is rather complete: It gives an overview of all nodes and all resources in the cluster. It also shows all configuration parameters related to these objects.

Figure 6.7. The Cluster Report utility gives a good overview of what is happening in the cluster.

[View full size image]

Apart from the Cluster Report utility, from the Cluster Manager window you also get a good overview of the current state of nodes and resources. If a node goes down, it will be detected in the next refresh of the utility and the node will be displayed with a cross in it. Note that this is considered a major event, and therefore the epoch number will be updated by one. Resources that have a problem are also clearly marked in the Cluster Manager overview.

From an administrator perspective, this screen is important because resource management can be performed from it. The following options are available:

  • Online: Use this option to bring a resource online that is currently offline.

  • Offline: If you need to do maintenance on a server and need to bring down a resource temporarily, this is the button to do so. It will disable the resource and no failover will occur.

  • Migrate: Another option to use when you need to bring down a resource temporarily is to migrate the resource to another server. Select the resource, click Migrate, select the server on which you want to run it, and the resource will be migrated automatically.

  • Respond to Alert: If a resource is in a Failback Alert, Failover Alert, or Start Alert state, use this option to troubleshoot. These modes are used when failover for resources is set to Manual. If, in that scenario, a cluster node has gone down, the resource associated with the given node will enter Failback Alert, Failover Alert, or Start Alert state, depending on its current state. As an administrator, you need to start it manually on another node in that scenario. Use the Respond to an Alert link to do this.

At the lower part of the Cluster Manager screen, you can configure the update interval of the screen. You can set it to anything between 2 seconds and 5 minutes. In most situations, the default setting of 30 seconds works fine.

Monitoring Cluster Events From the Cluster Event Log

The last part of the iManager Cluster Role is the Cluster Event Log task. As shown in Figure 6.8, this screen provides an overview of all recent events for the selected cluster.

Figure 6.8. The Cluster Event Log provides an overview of all events that have occurred since the log was last cleared.

[View full size image]

In the Cluster Event Log, browser buttons are provided to browse through large log files. In case too many events are listed, you can use the Filter button to specify a search filter. In that case, only events that match the criteria in the filter are displayed (see Figure 6.9). The categories in the filter are divided into four topics:

Figure 6.9. The Event Log Filter helps you in displaying just the information you need to see.

[View full size image]

  • Severity: Specifies the severity level. By default, events of the categories Error, Warning, and Normal are listed. Deselect any of these options to see fewer options.

  • Node: Displays all nodes in the cluster. By default, all nodes are selected. Deselect any to see fewer log entries.

  • Resource: Shows all resources in the cluster. By default, all are selected.

  • Start Date and Time: Specifies the date and time from which to display the first entries in the Cluster Event Log. Set this to anything you like to see log entries related to that particular date and time.

A filter that is set is not cleared automatically. If you want to display all information again, use the Clear Filter button. To remove all entries from the log file, you can use the Clear Log button.

Managing Cluster Resources on OES NetWare with Remote Manager

Although iManager is the interface to manage Novell Cluster Services, some management tasks are still available from Remote Manager. This works, however, only when accessing Remote Manager on an OES NetWare box; there is no Novell Cluster Services management Remote Manager module for OES running on Linux. In this section, the Remote Manager management options for Novell Cluster Services are briefly discussed. Because all options are exactly the same as the options mentioned in the section about iManager, no details about the specific options are provided in this section.

To access the management capabilities from Remote Manager, you have to access Remote Manager on your NetWare server. It can be accessed from a supported browser such as Mozilla Firefox or Internet Explorer at https://yourserver:8009. After you've logged in as a user with sufficient administrative permissions, Remote Manager will display all available options. The cluster-related options are under the Clustering heading, as shown in Figure 6.10.

Figure 6.10. All cluster-related management tasks in Remote Manager on NetWare are under the Clustering header.

[View full size image]

Under the Cluster Config option, you can find the cluster-related eDirectory objects, as well as an option to create new cluster volumes and resources. All objects that are listed are hyperlinked as well; if you need to modify some properties, you can do it from this interface.

The Cluster Management option can be used to display properties of selected nodes. It will show information about the availability of nodes and resources. This information is refreshed automatically, every 10 seconds by default. The master node in the cluster is marked with a yellow dot in its server icon, as shown in Figure 6.11; nodes that are currently unavailable are indicated in gray. It is also possible to set resource properties and access the Cluster Event Log from this interface.

Figure 6.11. In Remote Manager on NetWare, the current status of cluster nodes and resources is displayed from the Cluster Management interface.

[View full size image]


Managing Novell Cluster Services from the Command Line

Management of Novell Cluster Services from the browser interfaces is convenient if it all works well. If it doesn't, you need tools that go deeper. Or sometimes you are sitting behind your server in the server room and you want to quickly perform a task such as migrating a resource to another cluster node. Therefore, in this section you will learn how to start and stop Novell Cluster Services from the command line and what commands can be used from the command line to manage the cluster. The command-line tools are rather similar on NetWare and Linux and are therefore covered in one section. There are separate sections about starting and stopping Novell Cluster Services on NetWare and on Linux.

Starting and Stopping Novell Cluster Services on Linux

The main file to start Novell Cluster Services on Linux is /etc/init.d/novell-ncs. This startup script is created automatically on system installation. Normally, it is called from the system startup procedure, but if for whatever reason you need to start and stop all Novell Cluster Servicesrelated modules manually, you can run the script from the command line. The command you need is /etc/init.d/novell-ncs stop to stop it, or /etc/init.d/novell-ncs start to start it. Be aware that before stopping Cluster Services, you should issue the cluster leave command to leave the cluster. If you want to stop Cluster Services on all nodes, use the cluster down command before unloading the software.

There is a symbolic link in /usr/sbin called rcnovell-ncs that can be used instead. So, as an alternative, you can type the short command rcnovell-ncs stop or rcnovell-ncs start.

Note

On Linux, all services can be started with the so-called start script in the directory /etc/init.d. The way to use these scripts is by entering the name of the script, followed by start, stop, or restart. In some but not all cases, some other options are supported as well. The disadvantage of this method is that the directory /etc/init.d is not in the search path of an OES Linux server. Therefore, on OES Linux a symbolic link (let's say a shortcut) has been created in the directory /usr/sbin, which is in the search path. The name of this symbolic link includes the exact name as it is in /etc/init.d, preceded by the letters rc. For example, /etc/init.d/novell-ncs would be /usr/sbin/rcnovell-ncs. Using these symbolic links as commands makes it much more convenient to start, stop, and restart a service manually. For example, the user does not need to use /etc/init.d/novell-ncs restart to restart the cluster service, but can just use rcnovell-ncs no matter what directory currently is active.

The rcnovell-ncs command has some other options as well:

  • rcnovell-ncs status: Can be used to display the current status of the Novell Cluster Services support modules. It will show whether NCS is currently in use.

  • rcnovell-ncs restart: Stops and starts the service.

When starting Novell Cluster Services manually by using this script, you should be aware that there are some system dependencies. In earlier versions of the software, there was a dependency that the Novell Storage Services (NSS) support modules needed to be loaded on OES Linux, but as of support pack 2, this dependency has been eliminated. Also, you should make sure that the shared storage needed for Novell Cluster Services to run (for example, to get access to the Split Brain Detector partition) is present as well. Make sure that any such dependencies are resolved before trying to load the software manually.

Starting and Stopping Novell Cluster Services on NetWare

On NetWare, it is the responsibility of one single NetWare Control File (NCF) to load Novell Cluster Services. The name of this NCF is LDNCS.NCF (short for Load Novell Cluster Services), and it is called automatically from AUTOEXEC.NCF. As for Linux, on NetWare as well you should be aware of the existing dependencies. There is no need to load NSS manually before you start Cluster Services, but there is, however, a need to activate all shared storage devices before you start. This is typically performed from the STARTUP.NCF file. To unload NCS, the ULDNCS.NCF file can be executed. This will automatically unload all cluster-related modules. Unloading the software will automatically have the node leave the cluster first.

Managing Novell Cluster Services from the Command Line

When tuning and troubleshooting Novell Cluster Services, the administrator has the important task of managing Cluster Services from the command line. For this purpose, the cluster command was created. This command has many options that can be used from the command line to manage the cluster environment. All the commands in the following list are arguments for the cluster command (for example, the command down should be used as cluster down). To indicate whether the command can be used on NetWare, on Linux, or on both, an (n), (l), or (n+l) is added after the name of the arguments:

  • alert {resource}{yes|no}: (l+n) This command can be used on a resource that currently is in an alert state. This happens when the resource start, fail over, or fail back mode is set to Manual and the resource is waiting for something to happen. If it is, use this command to specify the name of the resource, followed by yes if you do want the resource to start, fail over, or fail back, and followed by no if you don't want this. For example, use cluster alert dhcp yes to allow the DHCP resource to continue to do what it needs to do. You would typically use this option after querying the status of the given service with the cluster status command. For example, to query the status of the DHCP status, first use cluster status dhcp, and then if the resource is in an alert state, use cluster alert dhcp yes.

  • scan for new devices: (n) This command will scan for new devices on all NetWare nodes in a cluster. The command is equivalent to the NetWare command scan for new devices and can be used if you want the cluster to discover new Logical Units (LUNs), devices, volumes, or partitions on shared media.

  • convert {preview|commit}: (l) This command is used on Linux to finalize the migration from NetWare to Linux. After the software on the NetWare nodes has been replaced with Linux, all resources that were created on NetWare have to be converted to Linux in order to use the proper syntax. Use cluster convert preview to view what will happen. This will display the old syntax of the cluster load scripts and the new syntax by which it will be replaced. If you are happy with the new syntax, use cluster convert commit to commit the changes.

  • cvb rebuild: (n) On NetWare, this command can be used to disregard the current Cluster Volume Broker database. This will rebuild the database, which will allow the use of duplicate pool names. This command may be useful in case of problems activating and deactivating clustered volumes.

  • cvsbind {add, del}{resource}{IP address}: (n) On NetWare, this command is used to enable applications that rely on bindery to access a virtual server on the cluster. If used, you need to specify the resource name of the application and the IP address of the virtual server. Also use the add or del switch to enable or disable bindery access to the virtual server.

  • dhcp {context}: (n) On NetWare, this command can be used to specify the context in eDirectory where the NCS software should look for Dynamic Host Configuration Protocol (DHCP) configuration. This command can be used to troubleshoot problems loading the eDirectory-enabled DHCP server.

  • down: (l+n) This command removes all nodes from the cluster.

  • info {all, basic, notification, priority, protocol, summary}: (l) This is used to display (sometimes a lot of) information on the STDOUT. This command works on Linux only. The option all will show a lot of information. This information can be filtered by type when the basic, notification, priority, protocol, or summary options are used.

  • join: (l+n) This command adds the node where the command is issued to the current cluster.

  • leave: (l+n) This removes the node where the command is issued from the cluster.

  • maintenance {on|off}: (l+n) This puts a cluster node in maintenance mode. Maintenance mode will temporarily suspend the cluster heartbeat. This is useful if you need to reset the LAN switch without bringing all nodes in the cluster down.

  • migrate {resource}{node name}: (l+n) Use this command to perform a manual migration of a cluster resource. For example, use cluster migrate dhcp node2 to migrate the DHCP resource to the server called node2.

  • offline {resource}: (l+n) This command is used to bring a resource down. The command must be used on the node where the resource is currently active.

  • online {resource}{node name}: (l+n) This command starts the resource. If a node name is not given as an argument, a resource will start on its preferred node. If you want to start it on a specific node, the name of this node can be provided as an argument; for example, cluster online dhcp node2.

  • pools: (l+n) This can be used to display a list of NSS pools that are accessible by Novell Cluster Services.

  • resources: (l+n) This command shows a list of all resources that currently exist in the cluster, whether or not they are online.

  • restart {seconds}: (l+n) This command restarts Cluster Services software on all servers in the cluster after the number of seconds that can be specified as an option.

  • set {parameter}{value}: (l) Use this option to set cluster parameters for the cluster. All parameters that can be set from the graphical interfaces can be modified with this command. The following parameters are available:

    • IPADDRESS: Use this command to set the cluster IP address. Cluster software on all nodes must be restarted after changing this parameter.

    • PORT: This parameter is used to change the cluster port number.

    • QUORUMWAIT: Use this parameter to set the amount of time in seconds the cluster should wait before resources start to load.

    • QUORUM: This parameter defines the minimal number of nodes that must be running in the cluster before resources start to load.

    • HEARTBEAT: This option specifies the amount of time between heartbeat multicasts on the network.

    • TOLERANCE: Use this parameter to specify the amount of time the master node gives all other nodes in the cluster to indicate that they are alive.

    • MASTERWATCHDOG: This parameter specifies the number of seconds between packets that are sent to check whether the master is still alive.

    • SLAVEWATCHDOG: This specifies the time in seconds slave nodes will give the master to indicate that it is still alive.

    • MAXRETRANSMITS: Use this parameter to specify the maximum number of times the master should try to contact a slave node in the cluster.

    • ENABLEEMAIL: Use this parameter to indicate whether you want to set email notification on or off. This parameter takes three different options: OFF, CRITICAL, and VERBOSE.

    • EMAILADDRESSES: Specifies the email address where cluster alerts should be sent. More than one address can be included, if the addresses are separated by spaces.

    • EMAILOPTIONS: If this parameter is given the value XML, event notification by email will use the XML format. If no parameter is provided, the email will be sent in text only.

  • stats {display,clear}: (l+n) This shows the node number, node name, and heartbeat information on the log console screen. On NetWare, this information is displayed on the logger screen; on Linux it is displayed at STDOUT (which normally is the screen where the command was issued).

  • status {resource}: (l+n) This command reports detailed status of the specific resource. This also shows the number of times the resource has been migrated or failed over to another server.

  • view: (l+n) This command shows the node name, epoch number, name of the master node, and all other nodes in the cluster.


Logging, Event History, and Alerting

As an administrator of Novell Cluster Services, you need to know what happens with the system. Novell Cluster Services stores events in its Cluster Event Log. Another way to find out what went wrong is to check the messages displayed in iManager and Remote Manager. This is all good and well, but wouldn't you prefer to know immediately when a problem occurs? Email notification can be a solution.

To be able to use email notification, you must first set up the mail notification settings for Remote Manager. Novell Cluster Services uses this mail server setting to send emails through SMTP.

Note

All tasks that can be performed from Remote Manager can be done from iManager and sometimes with cluster console commands as well. For those interested in the working of Remote Manager, this procedure explains how to use Remote Manager on NetWare to set email alerting. One easy way of doing the same from an OES Linux environment is by using the cluster set enableemail command as described earlier.

Follow these steps on every node in the cluster to configure your mail server configuration:

1.
Open Remote Manager at https://yourserver:8009.

2.
Click the Configure icon in the top panel.

3.
Scroll down to the HTTP Interface Management section. Then click the link for Mail Notification Configuration. This brings up the screen shown in Figure 6.12.

Figure 6.12. Remote Manager has its own option to specify where email alerts should be sent.

[View full size image]

4.
Enter the IP address or DNS name of your outgoing SMTP server in the Primary Mail Server Name field. Click Save to store this data before continuing. Make sure that if you are using the DNS entry, all your cluster nodes can resolve that name to the correct IP address.

5.
You can enter an optional second mail server address and an identifier that will display where the message originated from. We suggest that for the latter option, you use the server name in that fieldfor example, SRV1@COMPANY.COM. This will help you identity where the problem occurred, and it can help you when you're setting up email rules to automatically organize these types of emails. Also, some relaying gateways and spam filters will block messages without a sender's address. Click Save to store this data before continuing.

After setting up the mail server configuration, you must specify where the messages will be sent. Cluster Services does not use the email addresses from Remote Manager for this purpose. You will have to configure the email addresses in the properties of the cluster object. In iManager, this is configured in the General tab of the cluster object properties page.

The Notification tab allows you to configure whether you want to receive notification of critical events only, or whether you prefer to receive all messages. The latter will send emails about everything that happens in the cluster. As an example, when you only migrate a resource to another node, an email notification will be sent to you. For day-to-day operations, you probably want to receive critical error messages only.

The messages are formatted with a simple clear-text message:

Subject: Cluster Resource Change
Resource: GW_SERVER State: Offline on Cluster Cluster01

The message format can also be configured as XML. When you change this setting in the cluster object properties, you will have to stop the cluster and unload and load the clustering software. The same Cluster Resource Change message that was displayed in normal text will look like this in XML format:

Subject: Cluster Resource Change -XML

<XML VERSION="1.0">
<CLUSTER-EVENT>
    <RESOURCE-EVENT>
        <RESOURCE-NAME>GW_SERVER</RESOURCE-NAME>
        <CLUSTER-NAME>Cluster01</CLUSTER-NAME>
        <CLUSTER-TIME>2006 2 8 16:21:33</CLUSTER-TIME>
        <EPOCH>0</EPOCH>
        <STATE-OFFLINE/>
    </RESOURCE-EVENT>
</CLUSTER-EVENT>
</XML>

The server that runs the master resource is the one that sends out the email messages. You never know which node will become the master node, so that is why the configuration of the mail server needs to be set up on every node in the cluster. When no emails are sent out and the configurations seem to be okay, on NetWare, you can enable the SMTP screen on the server that is the master node to check what the software is doing. On Linux, this option is not available. Use the following command to enable the screen:

SMTP SCREEN = 1

An extra server console screen will become active and will display all SMTP activity from the Remote Manager SMTP service. Setting the value to zero disables the screen.


Summary

In this chapter you have learned how to manage an NCS environment. Although different tools can be used, iManager really is the tool to use for your day-to-day NCS management tasks. As an alternative, the cluster command can be used to manage NCS from the command line, although this is a lot harder to work with. Only in an NCS NetWare environment, is Remote Manager available for management of NCS. For those still working in a NetWare-only environment, some procedures working from Remote Manager have been described as well.


Chapter 7. iSCSI

The topic for this chapter is not one purely related to clustering. iSCSI is a storage-related protocol that can be used for many storage needs, including shared storage for clustering. In this book we do not cover other storage techniques in depth, such as storage area networks (SANs) and Direct Attached Storage. But because Novell ships Open Enterprise Server (OES) with support for iSCSI and because it is a really useful technique for use with clustering, we have decided to add a separate chapter on this topic. Another technology for sharing storage across the cluster is Distributed Replicated Block Device (DRBD). This technology is covered in Chapter 11, "Using SUSE Linux Enterprise Server Clustering Options."

In this chapter we introduce what iSCSI is and we discuss some general concepts for this protocol. We also describe how to use the software with Open Enterprise Server for NetWare and Linux.


iSCSI Introduction

The iSCSI protocol offers a transport layer for Small Computer Systems Interface (SCSI) blocks over TCP/IP networks. Typically, local devices were and still are connected through SCSI . Such devices are disks and tape drives in the data center and, for example, scanners on the desktop. SCSI is a protocol that transports data by means of a communications channel called a bus, the SCSI-bus, where devices, including the host adapter in the computer, have a unique address and control blocks are being send back and forth between devices. For example, a server sends a disk write control block to a SCSI disk and the disk replies with a control block informing the server that the data has been written. iSCSI works as a wrapper around these control blocks called Command Descriptor Blocks (CDBs), encapsulating them inside iSCSI protocol packets, called Protocol Data Units (PDUs).

iSCSI originated in the late 1990s from research by mainly IBM and Cisco. The first proposal for the iSCSI standardization was RFC 3347 (Small Computer Systems Interface protocol over the Internet) from July 2002 that described the requirements and design considerations for iSCSI.

Many vendors have since worked on the standard, and a first draft of the iSCSI technical specifications was laid down in RFC 3720 (Internet Small Computer Systems Interface) from April 2004.

Tip

There are many RFCs that provide very good insight into the workings of iSCSI. The ones mentioned before are an example, but another one that provides very useful information is RFC 3721, which describes the naming guidelines for iSCSI. The easiest way to access RFCs is from the original RFC Editor's website at www.rfc-editor.org. The website contains a search facility in which a search for the iSCSI keyword provides access to all iSCSI-related RFCs.

The original design guidelines describe the following usage scenarios for iSCSI:

  • Local storage access, consolidation, clustering, and pooling (as in the data center)

  • Network client access to remote storage (for example, a "storage service provider")

  • Local and remote synchronous and asynchronous mirroring between storage controllers

  • Local and remote backup and recovery

For this book we focus on the use of iSCSI for local storage access, consolidation, and clustering. Let us define these terms before we go into more detail of how iSCSI is being used in the Open Enterprise Server environment. Figure 7.1 shows an example of a typical iSCSI storage network as we will describe it.

Figure 7.1. A typical iSCSI storage network.

Local storage access refers to the replacement of Direct Attached Storage for servers. Typically, servers have locally attached disks. These do not necessarily have to be internal disks. It is also possible that a server accesses a dedicated storage enclosure with disks attached via SCSI or Fibre Channel. iSCSI can replace these types of storage connections. New technologies, such as Blade Servers, can greatly benefit from using iSCSI when SCSI and Fibre Channel connections are not available.

The consolidation part of the iSCSI usage scenario is about creating a single point of storage that can be accessed by many systems. This central storage point is called a target in iSCSI terminology. One central storage provider can be used from NetWare, Linux, Windows, and other operating systems.

Finally, the clustering part of the usage scenario is most important for us in this book. An iSCSI target allows more than one server to connect to its storage. Because of that, it is possible to create shared storage that can be used by Novell Cluster Services (NCS).

iSCSI Advantages

There are many benefits to using iSCSI in comparison to more traditional storage protocols and relatively new technologies such as Fibre Channel. The most important benefits are listed here:

  • Low-cost hardware requirements: For an iSCSI implementation relatively inexpensive hardware can be used. With a server operating as an iSCSI target, its inexpensive IDE disks could be used to offer large amounts of storage to other servers. The network connections between the initiator and the target can be created with regular ethernet hardware. We do not imply that iSCSI is always a low-cost solution. For performance and reliability reasons, you can choose high-end SCSI disks, for example, or very fast network connections. The important thing about iSCSI is that you have the choice to use the hardware you want.

  • Use of existing hardware: With iSCSI you can create a storage target from an existing NetWare server and provide the storage to other existing servers. With this approach you can even build a full cluster environment without the need to buy additional hardware.

  • Easy, manageable storage solution: The entire implementation of iSCSI can be done with equipment and technologies that you are already familiar with: ethernet and TCP/IP. Compared to implementing a Fibre Channel SAN, iSCSI is like a walk in the park. The ease of use also makes it easier to learn how to work with your storage solution.

  • Extended storage range support: With traditional SCSI you are limited to locating the storage equipment close to the servers. Fibre Channel can stretch that to a maximum of 6 miles or 10km. With iSCSI the sky is the limit. If you can reach your destination with TCP/IP, you can place a storage target theretheoretically, of course, because in real life your capabilities will be limited by factors such as bandwidth, amount of storage traffic, and network latency.

  • Flexibility in capacity planning and growth: Existing limits for SANs for the number of disks that can be used or the number of physical connections to a Fibre Channel switch do not apply to iSCSI. It is relatively easy to attach new iSCSI targets to an existing network and add the new storage to the existing servers. Because of the use of a network protocol (TCP/IP), there is no need to physically attach equipment, sometimes with the need to bring a server down.

How iSCSI Works

iSCSI is a technology that encapsulates SCSI blocks in TCP/IP packets. The way this is set up is that the original application that is using the device on the other side of the iSCSI connection is unaware of this extra layer.

Figure 7.2 displays the typical communication flow for an application or operating system call that reads or writes from or to a SCSI device. The application uses the OS-specific system calls for that, and the driver communicates with the disk with SCSI Command Descriptor Blocks.

Figure 7.2. The regular SCSI disk communication process.

[View full size image]

When we add iSCSI to this setup, nothing changes on the application or operating system side. It still uses the regular operating system calls to communicate with a device, but instead of communicating with a SCSI device driver, it communicates with an iSCSI driver disguised as a SCSI driver. For NetWare this is the ISCSIHAM driver; for Linux it is a kernel module called iscsi (which you can check with the lsmod command). In other words, this new driver receives requests as a normal disk does, but sends them to a storage device through the network without the application knowing about that extension.

Figure 7.3 shows the communication flow for the iSCSI storage protocol. In this example an application writes to or reads from a disk that seems to be local. The iSCSI driver sends the request across the TCP/IP network, where the iSCSI target takes care of handling the request on behalf of the iSCSI initiator.

Figure 7.3. Disk access flow in an iSCSI storage network.

[View full size image]

Note

In Figure 7.3 we have explicitly displayed a physical disk and not necessarily a SCSI disk. This is because an iSCSI target partition on a NetWare server can be created on both a SCSI disk and an IDE disk.

It is important to understand that the server holding the target partition does not have local access to the data on the iSCSI partition. iSCSI is a disk access protocol and has nothing to do with file systems. It is the iSCSI initiators that will create a file system on the target disk. If the server holding the physical disk needs to access the data, the only way for it to do that is by starting an iSCSI initiator session with itself. It is not recommended for permanent production environments to operate a server that accesses its target as a local initiator.

iSCSI Naming Guidelines

The iSCSI specification defines the naming rules being used for iSCSI targets and initiators. We will look into the way these iSCSI Qualified Names (IQNs), as they are called, are created in the Open Enterprise Server environment.

An iSCSI Qualified Name (IQN) is constructed of the following components:

[View full width]
iqn.date.organizational naming authority:string defined by organization or local naming
  authority

Let us analyze this naming scheme with an example that will be used for an iSCSI target later in this chapter:

iqn.1984-08.com.novell:iscsi-0054c1d8-6241-da11-b2be-000c2921e430.oesnw3.srv.oes

The target can be identified as an iSCSI target by the IQN prefix. The next part is a date field. This is the date that the company, Novell in this case, has registered the name being used in the next part.

The organization name is the reversed DNS name of the company that provides the iSCSI target. It is possible for your own implementation of iSCSI solutions to use your own company naming, but everything you do in the OES environment will be named with Novell's identifier.

In compliance with the naming standard, a colon (:) is being used to divide the DNS naming part from the unique string that identifies the iSCSI target. Novell has chosen to use iscsi as an extra prefix, followed by a unique identifier for the iSCSI target or initiator and the full distinguished name of the server where the target is hosted.


Using iSCSI in an Open Enterprise Server Environment

Now that we have looked at the story behind iSCSI, it is time to look into how we can use this protocol in the Open Enterprise Server environment. We will start with some usage scenarios for iSCSI, and later in this section we will discuss the technical requirements and limitations.

iSCSI as a Server OSIndependent Storage Platform

First and foremost, iSCSI is a storage protocol providing access to storage via a network. This is really the most simple implementation. A server with an iSCSI target or a hardware iSCSI target provides access to disk storage, and iSCSI initiators can be used on servers with different operating systems to connect to the storage. Figure 7.4 shows a diagram of the most basic of iSCSI solutions. In this example the target server would contain more than one target to which every initiator can connect individually.

Figure 7.4. iSCSI in its most basic form.

[View full size image]

The great thing about iSCSI is that it is available on many platforms. In this book we focus on NetWare as a target for iSCSI storage networks, but there is also iSCSI target software available for other platforms, and specific hardware can be used to provide an iSCSI target. On the other side, support for iSCSI initiators is even bigger. For all major operating systems, software exists to connect to an iSCSI target. Because of that, iSCSI is also very useful in building a cross-platform storage solution. NetWare can be used as the central location for storage, and all platforms, including NetWare, Linux, and Windows, can connect to the available storage. And not only can servers access storage across the network, but also client computers can be configured to access iSCSI targets.

iSCSI as a Low-Cost Shared Storage Solution

The preceding paragraph describes iSCSI as a storage platform for multiple operating systems. In that example the NetWare 6.5 server would be servicing multiple clients, but they would all access their own target. You can compare that to a SAN where there are Logical Units (LUNs) assigned to servers. But iSCSI can also be used to create a shared storage solution for clustering use. Figure 7.5 shows an example of a simple setup with three servers running Open Enterprise Server, either NetWare or Linux, or even mixed, that connect to a shared iSCSI target.

Figure 7.5. iSCSI can be used to build a low-cost clustering solution.

[View full size image]

Building a cluster with shared storage via iSCSI can even be done with existing hardware. If your organization has the server hardware and networking equipment to build such a solution, the only things you would need to buy are clustering licenses. This is true if you are setting up a three-node cluster or larger because a two-node license is always included with Open Enterprise Server.

iSCSI for Remote Storage

Many organizations have used the available maximum cable lengths of Fibre Channel to make storage from a central location available to servers on other sites. This can be on the same campus but also somewhere else in town. With iSCSI, doing this has become even simpler because the existing TCP/IP infrastructure can be used to set up the storage networkgiven, of course, that the bandwidth is sufficient for your storage needs. Figure 7.6 shows an example of the setup for storage on a remote site through the use of iSCSI.

Figure 7.6. iSCSI can also be used to store data on a remote site.

[View full size image]

iSCSI for Offsite Storage Mirroring

With iSCSI it is simple to access storage on remote sites. And if we combine that with the mirroring capabilities of Novell Storage Services (NSS), it becomes simple to have an offsite online copy of active data available on a server. Figure 7.7 shows how such a setup could look.

Figure 7.7. With iSCSI data can be mirrored to a remote site.

[View full size image]

The mirroring part takes place at the NSS partition level. This means that the remote server that holds the target does not have access to the partition that holds the actual data. In case of a failure you could access the remote data from either the main site or the remote site by starting an iSCSI initiator connection to itself.

When the main site becomes available, the initiator connection that the remote server had with its own target can be stopped and the original server can be connected. The NSS mirroring will be activated to bring the disks to the same level.

iSCSI Design Guidelines

In the most simple and less expensive configuration, the iSCSI traffic is transported via the existing local area network. For servers servicing a small number of clients or when the storage demands are not very high, this might be a workable scenario. For larger environments we recommend using a separate storage network. One of the major problems in a single-network environment is that the data you store from a client is sent out by the server on the same network to the storage. Effectively, all data travels via the network twice if you use one network, and that will consume valuable bandwidth and have a negative impact on performance. Figure 7.8 shows an example of a separate network for iSCSI storage.

Figure 7.8. For improved performance use a separate storage LAN.

Note

When you're setting up a scenario like this, it is very important that IP routing be disabled on the servers. Clients should never be routed to a server through a server's storage connection to another server.

When you're choosing hardware for your storage network, we recommend that you use at least 1GB ethernet hardware.

Additionally, you can use netowrk interface card (NIC) teaming to have multiple network interface cards servicing the iSCSI storage connections. It is a rule of thumb to have 1GHz of processor power available for every 1GB of network throughput. So for a server with 2GB network adapters, you would need 2GHz of CPU power available on top of what you already need for your applications. Because the TCP/IP stack is SMP-aware, it is a good idea to have at least two CPUs of a minimum of 2GHz each available in your servers.

Something else that will help with improving the performance of your iSCSI storage network is to use a hardware iSCSI adapter. Such an adapter offloads the processing capacity needed from the regular processor because it uses its own dedicated special-purpose processor to handle the iSCSI communications. Host bus adapters (HBAs) that are supported on both NetWare and Linux are available from QLogic.

iSCSI Target Requirements

iSCSI targets can be NetWare 6.5 servers running iSCSI target software. Every NetWare 6.5 server comes with the iSCSI target software.

Note

For Linux there is no support from Novell for iSCSI targets on Open Enterprise Server. There are several projects available on the Internet that provide target software for Linux. For Linux kernels 2.4 and 2.6 there is a driver from the University of New Hampshire (www.sourceforge.net/projects/unh-iscsi). Another project is www.sourceforge.net/projects/iscsitarget. And as it goes with open-source projects, we suggest that you check the SourceForge website when you need to get started with the software from such a project.

There are also hardware devices available that can be used as an iSCSI target. They generally support LDAP and CHAP authentication, so they will work with NetWare and Linux initiators. When you are looking into purchasing such a device, we recommend that you check Novell's "Yes, tested and approved" program. You can access the website at www.developer.novell.com/nss. From the drop-down list of available hardware to search for, select iSCSI Storage Area Network (SAN). This will allow you to search for all devices that have been tested and proven to work with Novell software.

iSCSI Initiator Requirements

Every server running NetWare 5.1 or above can be used as an iSCSI initiator. The software can be downloaded from http://download.novell.com and is available by default on servers running NetWare 6.5 or above.

For Linux the initiator software comes from Cisco and is shipped with Open Enterprise Server. Later in this chapter we will explain how to install the software.

Besides software requirements, there are no special hardware requirements, but as we mentioned in the section "iSCSI Design Guidelines," it is a good idea to have multiple network adapters in your server and to use a separate LAN for the storage network. Also use 1GB ethernet to set up the storage connections.


iSCSI Target Configuration on OES NetWare

NetWare servers can run as iSCSI targets in an Open Enterprise Server environment. Open Enterprise Server Linux servers, at the time of writing this book, cannot. In this section we will discuss how to install the software and set up an iSCSI target on Open Enterprise ServerNetWare.

Installing the iSCSI Target Software

To use a NetWare server as an iSCSI target, you must have the iSCSI target software installed.

Tip

The iSCSI initiator software is always installed with every NetWare server, starting with NetWare 6.5.

If you want to know whether the iSCSI software is already installed, you can check the installed products list from the GUI on the NetWare server console. Click Install from the Novell menu and a list of already installed products is displayed. If you would look at the file system, you would notice that the iSCSI target files (ISCSITAR.NLM) are always there. The iSCSI target installation really only creates the file with the credentials (LDAP Secret Store, being sys:\etc\iscsi.lss) and adds the load file to AUTOECEC.NCF.

If you install the iSCSI target, you will see the TON.NCF file being listed in your AUTOEXEC.NCF. This will load the target when your server boots.

There are three methods for installing this software:

  • You can select the component iSCSI Target during initial NetWare server installation with the custom installation method selected.

  • You can select a patterned deployment during NetWare server installation and select iSCSI Target as a possible deployment pattern. This will install only the necessary software for this server to function as an iSCSI target.

  • Using NetWare Deployment Manager (NWDEPLOY.EXE) from the NetWare operating system CD, you can add products, including the iSCSI Target software.

Creating the iSCSI Target

After installing the software, you must create an iSCSI target partition in the available disk space on your server. If you have never worked with iSCSI before, it is good to know that you cannot access the data in this iSCSI partition directly on the target server itself. It will be a real raw partition with no file system. Only after accessing the partition through the iSCSI initiator software can you create a file system and create and access data on the disk. There is no direct relation between your iSCSI target partition and the file system that will be created on it. For NetWare we will use NSS as the file system, but if you connect a Windows server to the iSCSI target, you can create an NTFS partition. So at this level of configuration, you should not think of the iSCSI partition as a file system but really as a raw disk partition.

You can create the iSCSI partition from the NSS Management Utility (NSSMU) at the server console. This configuration option is not available in iManager. Perform the following steps on the server that will hold the iSCSI target partition:

1. | Start the NSSMU and select Partitions from the main menu.
2. | Click Insert to add a partition on free space available. Browse to the disk with the free space where you want to create the partition and press Enter to select it. Tip If the disk is empty but no free space is displayed, you might need to initialize the disk. To do this, select Devices from the NSSMU main menu, select the device, and press F3 to initialize the disk.
3. | In the Select Partition Type dialog box, select iSCSI.
4. | Select the size you want to use for your iSCSI partition. Assigning a label is optional. We recommend that you assign a label if you are planning to create more than one target partition on this server. And because you never know what the future brings, always assign a label.
5. | Pressing Escape will return you to the Partitions screen, where the new iSCSI partition will be listed.
6. | Exit the NSS Management Utility.

You have now created the partition on the local hard disk that you want to share with iSCSI. It is now time to create the corresponding iSCSI target object in eDirectory. This object will be used to assign rights to initiators to allow them to access the target. The target object will automatically be created when the iSCSI target software is loaded. The command file that loads this software is in the AUTOEXEC.NCF file by default if the iSCSI target software is installed and is called TON.NCF. So we first unload the target software and reload it again to have the object automatically created.

Note

Keep in mind that stopping the target software will stop any running iSCSI sessions. So if this is an additional iSCSI partition, be sure to perform this task when iSCSI sessions can safely be interrupted.

Perform the following steps to create the iSCSI target object. We assume that the software was already loaded on this server so we start by unloading it:

1. | At the server console, stop the iSCSI target software with the TOFF command.
2. | Next, run the TON command to load the software. The object will now be automatically created.

The iSCSI target object will appear in eDirectory in the server container. You can see that in Figure 7.9. The object is named with the prefix iscsi followed by a global unique identifier. The last 6 bytes of the name, represented by 12 hexadecimal characters, is the MAC address of the server. In our example the name is iscsi-0054c1d8-6241-da11-b2be-000c2921e430 where 000c2921e430 is the server's MAC address.

Figure 7.9. The new iSCSI target object will be created in the server container.

[View full size image]

Note

If the object is not created, check the server's Logger Screen. If there are error messages about failed LDAP connections, re-create the LDAP authentication details following the steps in the "Troubleshooting" section at the end of this chapter and try again.

Now that the target object has been created, we will first introduce the iSCSI commands that can be used on the server. After that we will discuss how to connect servers, as initiators, to this new iSCSI target.

NetWare iSCSI Command Overview

You can manage iSCSI via the command prompt on the server or from Remote Manager. This section contains a listing of the most common commands. Remote Manager will be used in several parts of this chapter to show you how to work with that utility to manage your iSCSI environment. Table 7.1 lists ISCSINIT commands that can be used to create iSCSI connections; Table 7.2 contains ISCSI commands that can be used to configure the initiator itself; and Table 7.3 contains iSCSI target commands.

Table 7.1. Most Common ISCSINIT Control Commands

Command

Description

TON.NCF and TOFF.NCF

Starts and stops the iSCSI Target software

ION.NCF and IOFF.NCF

Starts and stops the iSCSI Initiator software

HELP ISCSINIT

Shows a listing of all available iSCSI commands

ISCSINIT DISCOVER networkaddress

Shows a listing of available targets at the specified target address

ISCSINIT CONNECT networkaddress [name]

Connects an initiator to all targets at the specified target address or to the target specified if the name of the target is provided

ISCSINIT DISCONNECT networkaddress

Stops the iSCSI session with the specified target

ISCSINIT INFO

Displays information about the active iSCSI sessions

Table 7.2. Most Common ISCSI Commands

Command

Description

ISCSI GET parameter name

Gets the current value of the specified iSCSI control parameter

ISCSI SET parameter name = value

Sets an iSCSI control parameter to the specified value

ISCSI LIST

Lists all iSCSI control parameters and their values

ISCSI REPORT

Generates an iSCSI system report (sys:\iscsireport.txt)

Table 7.3. ISCSITAR Commands

Command

Description

ISCSITAR SCAN

Scans for iSCSI targets on the server

ISCSITAR TARGETS

Lists all targets on the server

ISCSITAR SESSIONS

Displays all current iSCSI sessions to this server


iSCSI Initiator Configuration on OES NetWare

After the iSCSI target has been created on your NetWare server, you can configure the servers that will access the target. These "clients" are called initiators in iSCSI terminology.

The first step is to set up access rights for your initiators to be able to access the target. The easiest setup is when both the target and the initiator server are in the same tree. If they are not, you will need to set up LDAP authentication between them. Both methods are described in the text that follows. After configuring the security, we describe how to start the initiator connection.

Assigning Rights for Initiators in the Same Tree

Assigning access rights to an initiator in the same eDirectory tree is as easy as assigning the server as a trustee of the target object. It does not matter what rights you assign, just leave them as the defaults, as long as the object is listed as a trustee. To assign trustee rights from iManager, perform the following steps:

1.
In iManager select Rights from the Roles and Tasks panel.

2.
Select Modify Trustees and select the iSCSI target object.

3.
Click the Add Trustee button and select the server object for the NetWare server that will operate as the initiator. In a clustered environment assign all nodes that will access the iSCSI target as trustees. In Figure 7.10 you can see that two NetWare cluster nodes are assigned to the iSCSI target object that was created earlier in this chapter.

Figure 7.10. In a clustered environment add all cluster nodes as trustees of the target object.

[View full size image]

4.
Leave the rights at the defaults and click OK to save the configuration.

Assigning Rights for Initiators in Different Trees

Assigning rights to servers that are not in your tree is not possible with direct trustee rights because the servers do not have a local object you can assign as a trustee. Therefore, you must first create an object that references the iSCSI initiator to be used to assign the rights to allow access to the target.

Creating this object is not possible with iManager at the time of writing this book because that utility does not allow you to create objects for which it does not have a plug-in. ConsoleOne allows us to do this, so the following steps are based on using that utility. Any error messages about a snap-in not being available can be ignored because indeed there is no snap-in available. Here's what to do:

1.
Start ConsoleOne and browse to the iSCSI target container. We suggest that you create the initiator objects there.

2.
From the File menu select to create a new object and select iscsiInitiator as the object type. The name to use is the name of the server that will be functioning as the iSCSI initiator.

3.
From within ConsoleOne select the iSCSI target object and assign the newly created initiator as a trustee. When setting up a clustered environment, create an initiator object for every server that will become a cluster node and assign all initiator objects as trustees to the target object.

4.
On the initiator server set the InitiatorName to reflect the newly created initiator object. Use the following command to do this (note that this set command is case sensitive):

iscsi set InitiatorName=baseIQN:initiator_objectdn

The following command is an example of a server called OESNW4. It connects as an initiator called OESNW4 located in the OES container in the OES-TREE:

iscsi set InitiatorName=iqn.1984-08.com.novell:.OESNW4.OES.OES-TREE

Starting the Initiator Connection

The initiator software is always available on every NetWare server starting with NetWare 6.5. Connections can be started from the command line or from Remote Manager. We will discuss the commands here first. If you want to set up persistent sessions that will be started when the server boots, you must add the commands to the server startup file. You cannot accomplish that through Remote Manager.

Using Console Commands

The iSCSI initiator connection can be started from the command line with the following commands:

1. | Start the initiator software with the ION command.
2. | Connect to the target with the ISCSINIT CONNECT command.

The syntax of the ISCSINIT CONNECT command is as follows:

ISCSINIT CONNECT IP address [target name]

The basic usage of the ISCSINIT CONNECT command is to connect to all targets available on a host. In that scenario there is no need to specify a name for the target. The IP address is sufficient.

Note

If more than one target exists but this initiator has rights to only a single target object, it will see only the single target.

If you are configuring access to a server that has a single target, there is no need to add additional parameters. If more than one iSCSI target exists on a host and you want to connect to only one target, use the ISCSINIT DISCOVER command to find which targets are available. The command will also show you their iSCSI Qualified Names. Here's an example of an IQN:

iqn.1984-08.com.novell:iscsi-0054c1d8-6241-da11-b2be-000c2921e430.oesnw3.srv.oes

Specify this name to connect to the target with the discovered IQN.

If you want to use these commands to set up persistent connections, add them to your AUTOEXEC.NCF file. Also, do not forget to start the iSCSI initiator software from that file with the ION command.

Tip

It might be tempting to add your initiator commands to the ION.NCF file, but we suggest not doing that because these system-provided NetWare Control Files (NCFs) might be overwritten when you install an update to your server.

If you will be using iSCSI in a Novell Cluster Services installation, make sure that the initiator is loaded and sessions are started before the Cluster Services software is loaded. We suggest that you delay the loading of the clustering software for one to three minutes to let the server fully initialize. If your server is still very busy loading the operating system and has just set up an iSCSI connection to the storage, including the Split Brain Detector, the server could just be too busy to respond to cluster heartbeat packets and be thrown out of the cluster even before the party begins. Use the following commands to delay loading the cluster software after loading the iSCSI software:

ION.NCF
DELAY 60
LDNCS.NCF

Using Remote Manager

iSCSI initiator connections can also be started from Remote Manager:

1. | In Remote Manager expand the Storage Services link and select iSCSI Services. If this feature is not listed, the iSCSI software is most likely not loaded.
2. | Click the Add Target icon to add a session to an iSCSI target.
3. | Enter the IP address or hostname of the server that holds the target, and click Next. Note To be able to use the hostname here, the server that runs Remote Manager must be able to resolve that hostname, not the workstation that you use to access Remote Manager.
4. | A list of available targets will be displayed. Click the next button that appears near the target for which you want to set up a connection.

After this session is configured, it will be displayed in the Remote Manager iSCSI Services panel.


iSCSI Initiator Configuration on OES Linux

It was mentioned before that Open Enterprise ServerLinux supports only iSCSI initiators and no targets. In this section we explain how to connect to NetWare iSCSI targets, such as the ones created previously in this chapter.

The iSCSI initiator software does not work with eDirectory natively, as it does on NetWare. We will therefore have to set up an LDAP configuration to control access to the target. But first we will walk you through installing the software.

Installing the iSCSI Software on Linux

To work with iSCSI, you must install the Cisco iSCSI software package on your Open Enterprise Server. It is shipped by default with the SLES9 installation CD-ROMs. You install the software from YaST by performing the following steps:

1.
From YaST select the Software section.

2.
Select Install and Remove Software.

3.
For the filter type select Search from the drop-down list.

4.
Type iscsi as the keyword to search for, and click Search.

5.
The iSCSI package will be displayed in the panel on the right, as shown in Figure 7.11.

Figure 7.11. For Linux the iSCSI package must be installed from YaST.

[View full size image]

6.
After selecting the iSCSI package, click Accept to install the package. Provide the SLES CD-ROMs when requested.

Setting Up Access Control

For a Linux initiator to connect to a NetWare iSCSI target, it uses an LDAP name to connect to eDirectory. For this to work, the Linux server connecting to the server must be a trustee of the target object. Perform the following steps to assign the server as a trustee:

1. | In iManager select Rights from the Roles and Tasks panel.
2. | Next, select Modify Trustees and select the iSCSI target object.
3. | Click the Add Trustee button and select the Linux servers that will run as initiators. No special rights are needed. Just leave the default rights in place.

Configuring the Initiator Software

The server that will function as the initiator must be configured in the initiator configuration file. Open the /etc/initiatorname.iscsi file and modify the following line as we did for our example:

InitiatorName= iqn.1987-05.com.cisco:cn=oeslin1,ou=oeslin1,ou=srv,o=oes

Next you must configure the iSCSI software to connect to the target. Add the target information to the file /etc/iscsi.conf. Modify this line as for our example:

DiscoveryAddress=192.168.1.30

Connecting to the iSCSI Target

After the access control configuration and iSCSI software configuration have been completed, you can start the iSCSI session. Use the following command to start the session:

/etc/init.d/iscsi start

Ignore any messages to configure newly found hardware. Use the iscsi-ls command to verify the status of the connection:

iscsi-ls

This will display information about the connection to the target. To find information about the local device information that you will need when creating your Linux partitions or clustering configuration, use the sfdisk command:

sfdisk -l

With this command all local disks and their partitions are displayed, such as in the example displayed in Figure 7.12. In our example, device /dev/sdb is the one we will be using as the Split Brain Detector disk in an Open Enterprise Server solution.

Figure 7.12. Device /dev/sdb is the iSCSI target disk that holds the Split Brain Detector.

[View full size image]

For iSCSI sessions that will be used in a clustering or other permanent solution, it is important to start the software at boot time. Typically you want to do this for runlevels 3 and 5. These are the ones for which the ISCSI software will be enabled with the following command:

chkconfig iscsi on


Building an Open Enterprise Server Cluster with Shared iSCSI Storage

When you have gotten to this point in this chapter, you have read all there is to know about creating iSCSI connections. When you have your iSCSI connections in place, you can start the Novell Cluster Services installation. To help you keep the overview of what needs to be done, we have created a flowchart, displayed in Figure 7.13, that walks you through the configuration steps of setting up a cluster with shared iSCSI storage.

Figure 7.13. Follow these steps to build an iSCSI cluster.

[View full size image]


Advanced iSCSI Topics

In this section we will cover some additional iSCSI topics to make working with iSCSI in daily life easier and to help you fix any problems you might have.

Monitoring iSCSI Sessions

Active iSCSI sessions can be listed on the local server on both NetWare and Linux, but the most comprehensive view is available from Remote Manager. We will first show you two examples of how to view the information locally for both targets and initiators. Next we will look into using Remote Manager.

Status Information For Netware Iscsi Targets

On the server console you can use the iscsitar command to retrieve information about the available targets and sessions with the following commands:

iscsitar targets
iscsitar sessions

Status Information For Netware Iscsi Initiators

On a NetWare server that functions as an iSCSI initiator, you can use the iscsinit command to collect information. Use the command with the following syntax to retrieve information about active sessions with iSCSI targets:

iscsinit info

Status Information For Linux Initiators

On Linux servers you can display statistical information with the iscsi-ls command. No parameters are needed. Just executing the command will display the session information for active target connections:

iscsi-ls

Figure 7.14 shows an example of an iSCSI listing to a target on a NetWare server.

Figure 7.14. Use the iscsi-ls command to display session information on Linux.

[View full size image]

Using Remote Manager To Collect Session Information

For NetWare it is possible to collect information about active sessions for both the target and the initiator from Remote Manager. When you enable the advanced view, you can even see detailed information about network communications and iSCSI performance.

To display session information, start Remote Manager and select the Storage Services link. Next click the iSCSI Services link that is displayed. The frame on the right will display information about active sessions. Figure 7.15 shows an example of NetWare Server running a target. The session shows a green icon to the left, which means that it is an active session. Click this icon to collect more information about the iSCSI session.

Figure 7.15. Remote Manager can be used to display session details.

[View full size image]

To display statistical information about iSCSI sessions, perform the following steps from the iSCSI Services screen in Remote Manager:

1.
Click the Properties icon from the session overview page.

2.
The initiator properties screen allows you to select the following two check boxes that enable the controls on the main iSCSI session page:

  • Performance and Trend Graphs

  • Reports

3.
After selecting these check boxes, click Apply.

4.
Go back to the main iSCSI Services screen. Two new links will be displayed there that allow you to look at graphs of the iSCSI activity.

Solving iSCSI LDAP Problems

Creating the iSCSI target objects in your tree relies on secure communications being configured on the NetWare server. This is performed during the iSCSI target installation on the server. If the configuration is no longer valid, you can re-create the so-called LDAP Secret Store.

Tip

Doing this might also be necessary if you have created new server certificates. In that case, first export the server's self-signed certificate from iManager or ConsoleOne to the file sys:\public\RootCert.der before re-creating the LDAP Secret Store.

Reconfigure the LDAP security information on the target through Remote Manager by following these steps:

1.
Make sure that the iSCSI target software is loaded. Even if it displays an error message, leave it loaded; otherwise, the iSCSI controls will not show up in Remote Manager.

2.
In Remote Manager expand the Storage Services link and select iSCSI Services. If this feature is not listed, the iSCSI software is most likely not loaded.

3.
Click the LDAP icon on the right panel and perform one of the following steps:

  • If the LDAP Secret Store (sys:\etc\iscsi.lss) does not already exist, you are prompted for a administrative username and password. Log in as the Admin user to create the LDAP Secret Store.

  • It can also be the case that the LDAP Secret Store does exist and you want to recreate the information because LDAP authentication does not function properly (for example, error messages are displayed on the Logger Screen when loading the iSCSI target software). In that case, click Remove LDAP Configuration from the iSCSI LDAP panel. (This is in effect the same as deleting the file sys:\etc\iscsi.lss.) Next perform the step outlined in the preceding bullet point to create the LDAP Secret Store.

4.
Stop and start the iSCSI target software with the TOFF and TON commands.

5.
If you have created iSCSI partitions, the iSCSI target objects will now be created.


Chapter 8. Cluster-Enabled Applications

What is a cluster without applications? It's not much use to your end users. So in this chapter we will discuss how to work with some common applications that typically run on Open Enterprise Server (OES) clusters, either NetWare or Linux. In this chapter you will find GroupWise, iFolder, Squid, and Samba as examples. Why not more? There are two reasons for that. One is that if you understand how clustered applications work and you grasp the concept behind them, you will be able to cluster-enable most of your own applications. The second reason is that there are too many applications out there to cover all of them. We should also point out that detailed descriptions for many Novell applications are available on Novell's documentation website at www.novell.com/documentation.


Cluster-Enable or Not?

Before diving into the configuration of applications on an Open Enterprise Server cluster, we first want to discuss why you sometimes do not want to cluster-enable applications. Or at least we want to make you think about which applications need the high availability a cluster can give you.

A simple example is a DNS server. You could configure it to load on different nodes with some sort of shared configuration. But does that make sense if you can already have two DNS servers configured on your workstations for fault tolerance? It could be useful in some scenarios, but at least keep the nature of the beast in mind when looking at high availability for an application.

As another example, we will look into ZENworks for Desktops. For a long time the documentation of that product stated that it could be run on a cluster. That was only partly true because files for application distribution could be installed on a cluster volume and thus could have high availability. The same is true for group policy files that can also be installed on the shared storage of a cluster. The part about not being able to cluster-enable ZENworks components was all about the server-side applications, such as Automatic Workstation Import (AWI) and the imaging server.

The reason we bring up this topic is that on too many occasions we have seen clusters becoming overloaded with resources when some of them did not necessarily need high availability. But it is so tempting to make everything available at all times, isn't it?

To go back to our ZENworks example, let's take a look at the AWI component. Why do we need high availability for that application? The basic question you should ask yourself is "What if this service is not available?" That is the reason for clustering in the first place: to keep services available to end users. With the AWI service not being online, what happens? Already imported workstations will just not be able to contact the import server and will not update the workstation's information and logged-in user history list. The workstation object's identifier is stored in the local registry and will be used to connect to eDirectory to read policies and applications that are assigned to the workstation object. So the first of the two problems that could arise is that when you check the user object, you would not see the latest logged-in workstation, possibly affecting remote control if you go there through the user object instead of the workstation object. And the second problem, which could be more serious, is that new workstations would not be imported into eDirectory, possibly resulting in policies and applications not being deployed.

So is that a problem? That is up to you to decide. Every environment is unique and the high availability level, the service level, and the daily activity will differ from site to site. We hope that with this example we gave you a fresh look at application high availability and the reasons to install applications in a clustered environmentor not.


Clustering GroupWise

Of all applications that run in clustered environments, GroupWise is definitely the one that is used the most. Why? Very simple: Email has become the number-one application in every office. Take away people's email and they are unable to perform their jobs, or so they say.

We will look into configurations for GroupWise on NetWare and Linux in clustered environments. We expect that you have experience with GroupWise. We expect you to know what a POA is and what an MTA is, and how these components work on a noncluster-enabled server.

The basics of clustering GroupWise on a NetWare or Linux server are the same, so we will first describe how clustering works for GroupWise in general.

Tip

For the most detailed information on clustering GroupWise on Open Enterprise Server, including clustering the GWIA and WebAccess, we suggest that you take a look at the GroupWise books from the Novell Press GroupWise expert, Tay Kratzer, at www.taykratzer.com.

General GroupWise Clustering

All GroupWise components that you run on your servers are important, but when you look at high-availability concerns for your users, the Post Office Agent (POA) and, if applicable, the Webmail web server and agent are the most important. Without these components users will not be able to log in to their GroupWise environment at all. Other components, such as the GroupWise Internet Agent (GWIA), might be missed less. It could, for example, take longer before Internet email is delivered if the GWIA has been down. How high a level of high availability you will need to implement depends on the way your business works and how your users use their email application.

If we focus on the use of GroupWise within organizations on onsite desktops, clustering works like this: The POA is loaded on one of your cluster nodes, and the users are connected to the POA with their GroupWise client application. If a failover occurs on the cluster or when you migrate the POA to another node, the connection from the client to the POA is not available for a short period. When the POA is configured in the correct way with the setting that it runs in a clustered environment, the client software will use a longer timeout to contact the POA to give it time to fail over. After the POA is loaded on another node, the client reconnects to the POA without the user knowing that the failover has occurred.

Tip

The GroupWise client also has a caching mode. In that mode all email is also stored on the local hard disk and email delivery is processed in the background. With this mode users would never notice a failover, except that mail is not being delivered at that time, and they can start GroupWise without a POA being online. It might not be practical to enable this mode for all your users. Especially those who work on a different PC each day will not really benefit because it would have to cache their mailbox when they start GroupWise. Laptop users will benefit from using caching mode the most.

When you're using WebAccess to access GroupWise, high availability is a totally different scenario if you would use the same clustering paradigm. In that case there is no client application that works for the user and could reconnect. It all happens in the browser and the session is kept alive on the server. This would mean that users will notice a failover because they would have to re-log in. A WebAccess high-availability scenario would consist of redundant or clustered WebAccess agents and high availability for the web servers through network load balancing. More on this topic can be found in specialized GroupWise clustering books, such as those mentioned in the beginning of this section from Tay Kratzer, at www.taykratzer.com.

Groupwise Resources Explained

The GroupWise components, such as the POA and Message Transfer Agent (MTA), run on a server that has access to the cluster shared storage. Therefore it is important that the software is loaded on the server where the shared storage is activated. If you have a separate resource for your cluster-enabled pool and another one for GroupWise, you can never make sure that they are active on the same node. There is no such thing as combining or chaining cluster resources to become dependent on each other.

The way to make sure that the GroupWise software is being loaded on the server where the shared storage is active is to add the GroupWise load commands to the load script of the cluster pool resource, where the NSS pool is activated and the volumes are mounted or where any other form of Linux storage is activated. And of course to add the unload commands to the unload script.

This also means that when you want post offices to be able to run on separate servers, for example to balance your load, you must create separate volumes where the post office directories can be located and from where the correct GroupWise agents can be loaded.

Groupwise Addressing in a Cluster

On a standalone server you could just install GroupWise with all the defaults on the server's primary IP address and on all the default TCP ports. For a cluster you need at least secondary IP addresses that can be used with the cluster resource to load the agents. With older versions of GroupWise there were issues in which the Post Office Agents would bind to all available IP addresses on the default port (1677). That posed a problem when running multiple post offices on a single server trying to listen on the same port on all the same IP addresses. With newer versions of GroupWise, a new feature has been added that allows you to bind an agent to a specific IP address. That is great, of course, because then a post office uses only that address, and if you leave it at the default port it will not conflict with other agents because they have their own address, where they can use the same port.

It is, however, good practice to still use separate port numbers for the agents you have running that might ever run on the same server. By not only binding to the specific IP address but also specifying a unique port number to all agents, you prevent future problems when you bring an agent online without configuring the specific IP address binding.

Clustering GroupWise on NetWare

Before the GroupWise installation is started on the cluster, the shared storage must be created. For the instructions in this section, we will install a GroupWise system with one domain and two post offices. The domain's MTA will use a separate storage pool and volume for the domain data, and the two post offices each will also use a separate pool for the post office directory. Therefore we require three cluster resources to store the data. Table 8.1 shows an overview of the GroupWise system we will create in this section.

Table 8.1. Overview of the GroupWise NetWare Cluster Environment in This Section

Component

Pool and Volume (also Agent Name)

Cluster Virtual Server

IP Address

Ports Used

Domain MTA

GWDOM1

GWDOM_Server

10.0.0.60

Message transfer port 7100 HTTP port 7180

Post Office 1

GWPO1

GWPO1_Server

10.0.0.61

Client/server 1677 Message transfer port 7101 HTTP port 7181

Post Office 2

GWPO2

GWPO2_Server

10.0.0.62

Client/server 1678 Message transfer port 7102 HTTP port 7182

So the preparatory steps that need to be taken are to create the cluster resources for the three NSS pools with their volumes. How to do this is described in Chapter 5, "Creating Cluster Resources."

Before starting the GroupWise installation, also make sure that the three resources are up and running and that you have access to all three shared disks. The best way to do this is to create a drive mapping to these three disks. With the three drive mappings available, you can make sure that the installation program can copy files to the volumes and create the directories for you.

Because installing GroupWise on a cluster is not too much different from installing on a real server, we expect that you understand the other necessary requirements needed for installing GroupWise, such as having rights to the tree root object to extend the schema if necessary.

Installing the Groupwise System on Netware

In general, there are two ways to create a new GroupWise system. The first method is by the book: Start from the installation CD and follow directions. The second method starts halfway down the track and is preferred by most experienced GroupWise administrators. The step in the default installation that requires the most time is copying the software to the software distribution directory. Therefore many administrators just copy the GroupWise snap-ins to their ConsoleOne directory and select the option New System from the GroupWise Utilities menu under Tools. This is also the step where our installation starts. If you want to go by the book, just start the GroupWise installation from the CD-ROM, and when you reach the step where it starts ConsoleOne, pick up from this point onward.

Note

The steps in this book are based on a GroupWise 7 installation. If you need to set up a GroupWise 6.5 system, you can follow the same directions because the setup instructions are identical.

ConsoleOne will be started automatically from the GroupWise installer, or you can start ConsoleOne yourself and perform the following steps:

1.
In ConsoleOne from the Tools menu under GroupWise Utilities, select New System.

2.
Specify the path to the software distribution directory. In our case we kept it simple and just specified the path to the software CD-ROM in our CD-ROM drive, being F:\. If you perform the full installation, the best choice is to install the software directory to a clustered resource to always have this directory available.

3.
Specify the eDirectory tree where the GroupWise system is going to be installed.

4.
Select to extend the eDirectory schema.

5.
Provide a system name for your GroupWise system. This name cannot be changed later so select a general name, not using your company name or anything else that might change. We have often used GroupWiseSystem as a generic name.

6.
Provide the name of the primary domain, in our case GWDOM1.

7.
Specify the domain directory, in our case GWDOM1:\GRPWISE\GWDOM1. This is a directory in the shared pool that was created as the shared resource for the domain.

8.
The next steps are general for GroupWise installations and not different for a cluster, so we only summarize the selections you need to make:

  • Domain context

  • Domain language

  • Domain time zone

9.
Specify the name of the first post office to be created, in our case GWPO1.

10.
Specify the path where the first post office will be created, in our case GWPO1:\GRPWISE\GWPO1.

11.
The next steps are general for GroupWise installations and not different for a cluster, so we only summarize the selections you need to make:

  • Post office context

  • Post office language

  • Post office time zone

12.
Next you must specify the type of link that will be used to access the post office from the MTA. In a clustered environment this must always be a TCP/IP link. In our case we link to IP address 10.0.0.61, client/server port 1677, message transfer port 7101, and HTTP port 7181. See Figure 8.1 for an example of the configuration of our setup.

Figure 8.1. Post office links must always be TCP/IP links.

[View full size image]

13.
For the Message Transfer Agent specify the IP address of the cluster resource where the domain directory is installed. In our case this is the GWDOM1 pool, which has IP address 10.0.0.60. The message transfer port we will use is 7100 and the HTTP port is 7180.

14.
At this point you can add users to the GroupWise environment. This can also be performed when the system is up and running. But be sure to create at least one user at this point or at a later time to be able to test the GroupWise system after installation.

15.
The GroupWise Setup Progress overview window will now show that it is ready to create the system. A summary of all information you have entered will be displayed. You can now start the creation process of the GroupWise system. If all components were successfully created, you can proceed to the next step.

16.
The GroupWise Setup Progress overview window will display the next step: installing the agent software. Start this process by choosing Next. If you ever have to start this step manually, as we will do for the second post office later, you can start the agent installation with install.exe from the agent software directory on the CD-ROM or in the GroupWise software directory on your system.

17.
After selecting NetWare as the platform for your agents, you must specify the system software installation directory. We advise you to install the software on the shared disk system. We start with creating a SYSTEM directory on the GroupWise domain pool resource GWDOM1. In our example in Figure 8.2 drive letter I: is mapped to the GWDOM1 volume.

Figure 8.2. Install the software on the shared disk, mapped to I: in this case. And also configure the agents for use with clustering software.

Note

Also do not forget to enable the check box to configure the GroupWise agents for clustering.

18.
At this time the GroupWise system is created on your NetWare cluster. There are still a few configuration steps that need to be performed. In the next paragraph we will explain what needs to be configured on the newly created system and provide short instructions on how to configure this in your environment.

One of the first steps to perform is to configure the load files that are going to be used to start the agents. The default load file is GRPWISE.NCF. This will be created in the default SYSTEM directory that was created in step 17 in the preceding instructions. Using this load file will not work because it would try to load the MTA and the POA from the same volume, but because we have split them on different pools and volumes, the agents would not be able to access their files.

The files we need to create are called GWDOM1UP.NCF and GWDOM1DN.NCF, and GWPO1UP.NCF and GWPO1DN.NCF. With these files we can start and stop the MTA and POA individually. We then add these NCF files to the cluster resource load and unload scripts. You could start by copying the GRPWISE.NCF to the four filenames we have just listed or creating them from scratch. The filenames we have selected here match pretty nicely with the files called GWDOM1.MTA and GWPO1.POA that were created for the agents.

Tip

Using these load and unload command files is better than adding the commands to the load scripts. Doing so allows you to stop and start GroupWise without accessing the cluster software; it allows you to delegate GroupWise administration and cluster administration to different administrators; and it keeps the load scripts short and simple.

The location of these files can be the SYSTEM directory on the shared volume, or the GroupWise domain and post office directory of the resource that the configuration files belong to. We choose the latter location because this will keep the SYSTEM directory generic and it will be very easy to find the correct files for every resource. If you choose to also do this, do not forget to copy the MTA and POA configuration files that were created to the same directory too.

Here are the paths and contents of the files in the setup we have created in this section:

  • GWDOM1:\GRPWISE\GWDOM1\GWDOM1.MTA
/home-GWDOM1:\grpwise\GWDOM1\

    (This file contains more lines, but this is the only relevant one here.)

  • GWDOM1:\GRPWISE\GWDOM1\GWDOM1UP.NCF
LOAD Restart Address Space = GWDOM1 GWDOM1:\SYSTEM\GWMTA @GWDOM1:\GRPWISE\GWDOM1\GWDOM1.MTA

  • GWDOM1:\GRPWISE\GWDOM1\GWDOM1DN.NCF
UNLOAD Address Space=GWDOM1

  • GWPO1:\GRPWISE\GWPO1\GWPO1.POA
/home-GWPO1:\GRPWISE\GWPO1\
/cluster

  • GWPO1:\GRPWISE\GWPO1\GWPO1UP.NCF
LOAD Restart Address Space = GWPO1 GWPO1:\SYSTEM\GWPOA @GWPO1:\GRPWISE\GWPO1\GWPO1.POA

  • GWPO1:\GRPWISE\GWPO1\GWPO1DN.NCF
UNLOAD Address Space=GWPO1

If you look at the preceding listing of the files, you'll notice that we use full pathnames throughout the entire setup. We do not like to be dependent on relative paths or search paths being available. It might take some more time to enter all paths, but it makes the system more robust.

Testing The New Groupwise System on Netware

The next step you need to take after creating all configuration and startup files is to copy the SYSTEM directory from the shared storage directory that you have specified during installation to the volumes of the other resources. You'll do this first to the volume for post office 1, GWPO1, and also to GWPO2. We will configure the rest of that post office later.

After you've copied the SYSTEM directory, it is time to start the newly created GroupWise system. Execute the load scripts manually to start the MTA and the POA on the cluster node where the resources are loaded at that time:

GWDOM1:\GRPWISE\GWDOM1\GWDOM1UP.NCF
GWPO1:\GRPWISE\GWPO1\GWPO1UP.NCF

If the GroupWise agents run as expected and your GroupWise system works as you want it to, just as you are used to in a standalone environment, unload the modules again with the unload scripts:

GWDOM1:\GRPWISE\GWDOM1\GWDOM1DN.NCF
GWPO1:\GRPWISE\GWPO1\GWPO1DN.NCF

Be sure to stop the MTA and POA before configuring the load and unload scripts.

Configuring The Cluster Load and Unload Scripts

It is now time to add the load and unload command files to the load and unload scripts of the cluster resources:

1.
In iManager open the Clusters role and select the Cluster Options task. Browse to the cluster object where the resources that need to be configured are located.

2.
Select the domain resource object and click the Properties link.

3.
In the Scripts tab add the load command file to the load script, as in the example in Figure 8.3. Add this to the end of the script.

Figure 8.3. Add the load command file to the cluster load script.

[View full size image]

4.
Also add the unload command file (GWDOM1DN.NCF in this example) to the unload script. In that script the command must be added before any of the other lines because first the GroupWise software must be stopped and only after that can the IP address be unbound and the volume and pool deactivated.

5.
Also configure the post office load and unload scripts.

After configuring all the cluster resources, you must bring them offline and back online again to start reading the new load and unload scripts. This is one of the advantages of using NCF files inside the cluster resource scripts: The scripts remain static. When you modify the NCS files on the file system, you can execute them manually or migrate the resource to another node. There is no reason to bring the resource offline for changes.

The Final Test: Migrate the Cluster Resources

Now that all the files and the cluster resources have been configured, it is time to perform the final test. Perform a manual migration of the cluster resources from the first node where they are started to all other nodes where they are configured to run consecutively. Perform this test step by step, resource by resource, and keep track of how the agents are loading on all the nodes. Pay special attention to check whether they are loading without errors, and check their connectivity with the other GroupWise components.

Adding a Post Office to the Existing Groupwise System on Netware

In the previous paragraphs we explained in detail how to set up a default GroupWise system. A minimum system contains one domain and one post office. We will now describe how to add a post office to an existing GroupWise system in a NetWare cluster (for this step we expect that you have already created the shared Pool and volume resource where the data for this post office will be created):

1.
In ConsoleOne select the container where the new post office object needs to be created. Select to create a new object and choose to create a GroupWise post office.

2.
In the Create Post Office dialog box add the post office name and provide the path to the shared volume where the post office database will be installed. In our example in Figure 8.4, this will be GWPO2 in the GWPO2:\GRPWISE\GWPO2 directory.

Figure 8.4. Configure the settings for the new post office.

3.
In the next dialog box you must set up the link for the new post office. Just as with the first post office this must be a TCP/IP link. The IP address, as we planned in Table 8.1, will be 10.0.0.62 and the port numbers will be as shown here:

  • Client/server port 1678

  • Message transfer port 7102

  • HTTP port 7182

4.
Click Finish to create the new post office.

5.
Copy the SYSTEM directory from one of the other GroupWise resource volumes to the volume for this new post office.

6.
In the post office directory create the new post office load, unload, and configuration files. You could run through the installation process for the agents by running the install.exe. But if you just copy the configuration files from the first post office and modify them with the correct information for this second post office, that is much easier. Here are the contents of the files and their paths for the new post office:

  • GWPO2:\GRPWISE\GWPO2\GWPO2.POA
/home-GWPO2:\GRPWISE\GWPO1\
/cluster

  • GWPO2:\GRPWISE\GWPO2\GWPO2UP.NCF
LOAD Restart Address Space = GWPO2 GWPO2:\SYSTEM\GWPOA @GWPO2:\GRPWISE\GWPO2\GWPO2.POA

  • GWPO2:\GRPWISE\GWPO2\GWPO2DN.NCF
UNLOAD Address Space=GWPO2

Note

The reason we have used the separate address space for the first and second post office is that you cannot unload the regular GWPOA.NLM for a specific post office. Running the unload script for a post office would effectively unload all post offices. By specifying an address space, we can now unload a specific post office.

7.
After creating the files, test the new post office by running the new load and unload command files. Be sure to stop the Post Office Agent before configuring the load and unload scripts.

8.
In iManager modify the second post office cluster resource load and unload scripts and add the load and unload command files the same way you did for the first post office.

9.
Now bring the resource offline and back online again to start the second post office in the system.

Clustering GroupWise on Linux

Before you start the GroupWise installation, there are a few preparatory steps you must take. One of them, for example, is installing ConsoleOne to the system where you will be installing GroupWise. Other requirements for GroupWise 7, such as rights to the eDirectory tree to expand the schema, can be found in the GroupWise 7 documentation and are no different for installing GroupWise on a cluster (and thus are not covered here).

For the GroupWise 7 installation that we will walk you through in this section, we expect that you have configured your cluster with a shared resource to install your GroupWise systems databases. Also, you will need to have shared storage available, such as is displayed in Figure 8.5.

Figure 8.5. An overview of the partitions created with EVMS for our GroupWise system.

[View full size image]

Note

Although Novell Cluster Services (NCS) depends on the Novell Storage Services (NSS) file system for services such as file sharing, there is no need to work with NSS for GroupWise. Because GroupWise is a true client/server application, there is no need for end user rights to be assigned to the file system. Other features, such as journaling, that NSS offers are important but are also offered by traditional Linux file systems, such as ReiserFS and ext3. Our opinion on using NSS is that you should do so only if you need the features offered by it and to use the other available file systems when those offer sufficient support. For our GroupWise cluster example we will use ReiserFS because this is SUSE Linux's preferred file system and it performs better than ext3 when using many small files, which GroupWise does. One last remark here: Although NSS is not a requirement for GroupWise, you will still find that it is a software requirement for NCS that NSS be configured on the server, even though you might not be using it.

The GroupWise system we will be creating in this section is very much like the system we created in the preceding section for NetWare. The naming standards, locations, and IP addressing are listed in Table 8.2.

Table 8.2. Overview of the GroupWise Linux Cluster Environment in This Section

Component

EVMS Volume

EVMS Mount Point

IP Address

Ports Used

Domain MTA

gwdom1

/grpwise/gwdom1

10.0.0.66

Message transfer port 7100 HTTP port 7180

Post Office 1

gwpo1

/grpwise/gwpo2

10.0.0.67

Client/server 1677 Message transfer port 7101 HTTP port 7181

Post Office 2

gwpo2

/grpwise/gwpo2

10.0.0.68

Client/server 1678 Message transfer port 7102 HTTP port 7182

Preparing the Linux Cluster to Install Groupwise

We start with preparatory steps on our cluster before installing GroupWise. We want to have the shared disks and resources configured so that the shared data store is online and tested.

For the system where we are installing GroupWise, we have already created EVMS volumes that can be mounted as /grpwise/gwdom1 and /grpwise/gwpo1. How to create a shared EVMS volume is explained in detail in Chapter 5. Perform the following steps to prepare your cluster for the GroupWise installation:

1.
First create a cluster resource for the GroupWise domain and post office. This is where we will install the first GroupWise components during installation. How to create a resource is explained in detail in Chapter 5. We will give you only the quick tour here. In iManager under the cluster options for your Linux cluster, select New to create a new resource and select the Generic_IP_Service template to create a resource. We have used the names GWDOM1, GWPO1, and GWPO2 for our example. Figure 8.6 shows an example of the most important lines from the cluster load script. Change your load script and unload script lines accordingly. We will explain how some of the commands need to be interpreted to modify the scripts to meet your needs. Note that the mount command in iManager might not fit onto one line in the screenshot, but it is really all on one line.

Figure 8.6. An example of a GroupWise post office load script.

[View full size image]

One other thing to note is that you must uncomment the lines in the scripts that start and stop the cluster service. You will configure this later, and we will explain how, after GroupWise has been installed.

2.
For each GroupWise component that will run on separate cluster nodes from other GroupWise components, you will need a separate cluster resource. The type of resource is different for a GroupWise Linux cluster as compared to the NetWare cluster that was configured earlier in this chapter. This is because on NetWare the GroupWise load scripts are executed from the cluster volume resources and on Linux we do not use those types of resources when using a Linux file system. Instead we use a generic resource, as explained in step 1, to mount the file system. Figure 8.7 shows an example of a Linux cluster with one GroupWise domain and two post offices. For each of these GroupWise components, there is a separate EVMS volume and a cluster resource.

Figure 8.7. Our starting point for the GroupWise cluster installation.

[View full size image]

Installing Groupwise on the Linux Cluster

When the volumes and resources are created, tested, and online on one of the nodes, it is time to start the GroupWise installation on that node. The full documentation of this installation is available from Novell's documentation website at www.novell.com/documentation. Therefore we will discuss only the major installation steps. To start the installation, run the install script from the GroupWise CD-ROM, for example, with this command:

/media/cdrom/install

Tip

If ConsoleOne is not yet installed, decompress the ConsoleOne tar-archive from the ConsoleOne directory on the CD-ROM and run the c1-install script from the extracted Linux directory.

All these installation steps, including the steps for testing the agents, are performed on the cluster node where the GroupWise resources are loaded and thus where the volumes are mounted and IP addresses are bound. Only after the installation and configuration are finished will we expand the GroupWise system to the other cluster nodes. Perform the following steps to install GroupWise:

1.
From the Novell GroupWise installation screen, select the option Create or Update a GroupWise System. Go through the mandatory steps of accepting the license agreement and such.

2.
The choice you will have to make is for the location of the software distribution directory. The default path is /opt/novell/groupwise/software. We suggest that you place this directory on one of the shared disks. In our case we have chosen to place it in the /grpwise/gwdom1/software directory. This way it will always be available on a running cluster node. Install at least the GroupWise agents in the software distribution directory and select the remaining software that fits the needs of your installation.

3.
After the file copy has finished, select to create a new system and let the installation start ConsoleOne to start the system creation utility. When asked for the Linux mount point, provide the path to the GroupWise installation CD-ROM for example, /media/cdrom.

4.
In ConsoleOne log in to eDirectory and select to install a new system from the ConsoleOne Tools menu in the GroupWise Utilities submenu.

5.
Go through the default GroupWise configuration steps for creating a new system. Provide the path to the software distribution directory and specify a GroupWise system name.

6.
Provide the domain name and domain directory as specified in Table 8.2. For our example this is gwdom1 and /grpwise/gwdom1.

7.
Provide other domain details such as the location of the domain objects, the domain language, and the time zone.

8.
Next provide a name and path for the post office. In our example this is gwpo1 and /grpwise/gwpo1.

9.
Provide other post office details such as the location of the post office objects, the post office language, and the time zone.

10.
The next option is to configure the post office link. Select to use a TCP/IP link and provide the secondary IP address of the post office resource, in our case 10.0.0.67. Also specify the client/server port, message transfer port, and HTTP port. In our example these are, respectively, 1677, 7101, and 7181. (See Table 8.2.)

11.
Next specify the domain's MTA network address information. In our case the IP address is 10.0.0.66, the message transfer port is 7100, and the HTTP port is 7180.

12.
Optionally add users to the new GroupWise post office. This can also be performed when the system is up and running. But be sure to create at least one user at this point or at a later time to be able to test the GroupWise system after installation.

After this step, a summary will be displayed from which you can review the information you have provided. If everything is okay, select Next to create the new GroupWise system. When the system is created, a message will be displayed that the system was successfully installed. Select Next to continue and start the agent installation.

Tip

If the agent software installation does not start automatically after you've created the GroupWise system, you can start that installation from the GroupWise installation screen by selecting the option Install Products. The installation can always be restarted with the install script from the software distribution directory.

Note

If you start the installation manually, do not forget to go through two steps: installing the agents and configuring the agents. This is different from what you were used to with GroupWise on NetWare.

13.
Go through the steps of configuring the GroupWise agents. If you have worked with GroupWise before, you can see in Figure 8.8 that the steps are identical to those in a GroupWise installation on NetWare. Add the first domain and post office to the configuration.

Figure 8.8. Add the first domain and post office to the GroupWise agent configuration.

[View full size image]

14.
When the configuration is complete, the default setting is to let the agents be configured to load at system startup. Uncheck this check box because we want to have the agents loaded by the clustering software.

15.
The configuration files for the MTA and POA are stored in this location: /opt/novell/groupwise/agents/share. The files are named with the name of your domain and post office and will have the mta and poa extensions.

Testing the New Groupwise System on Linux

It is now time to test the GroupWise agents to test whether they are working correctly. Start the agents with the --show parameter to display the window.

Later they will run as daemons without output and they can be accessed through the http interface. Start the agent software from the /opt/novell/groupwise/agents/bin directory and provide the names of the agent configuration files, for example, with these command lines:

/opt/novell/groupwise/agents/bin/gwmta --show @gwdom1.mta &
/opt/novell/groupwise/agents/bin/gwpoa --show @gwpo1.poa &

Tip

If the MTA shows the post office as closed, one of the most common problems is that the database version has not been set to version 7.You can check this in the post office object in ConsoleOne. If the database version shows as version 5.5, either just wait, or select the post office object and from the GroupWise Utilities menu select Mailbox/Library maintenance and run the default check. After that the database version should be listed as 7.

If the agents are working correctly, close them from the GUI.

It is now time to test the agents running as daemons. The main GroupWise script is called grpwise and is in the /etc/init.d directory. Check the status and start and stop all the agents with the following commands:

/etc/init.d/grpwise status
/etc/init.d/grpwise start
/etc/init.d/grpwise stop

Testing whether the agent daemons are working correctly can be done by accessing the HTTP port of the agent. Point your browser to the secondary IP address of the agent and add a colon and the port number. For our MTA listed in Table 8.2, this would be the following URL: http://10.0.0.66:7180.

Configuring the Groupwise Agents and Cluster Resources on Linux

At this point install the agents on each node in the cluster.

Note

On NetWare we can install the GroupWise software on the shared storage so that we have to install it only once. Because of the architecture of Linux, with its configuration files and daemons that the operating system and software itself expects to be in a default location local to the server, we cannot install the GroupWise software on the shared storage on a Linux cluster. Therefore, we have to install the software on each node.

Use the install script from the GroupWise CD-ROM or the one that can be found in the GroupWise software distribution directory. Follow these steps to perform a structured installation and configuration:

1. | Migrate the cluster resources to the second node. This will mount the volumes on that node.
2. | Start the GroupWise agent installation and also select Configure the Agents. Configure the agents for the domain and post office. Also do not forget to disable the check box to load the agents at system startup.
3. | Migrate the resource to the third node and all other nodes, and install and configure the agents until they are installed and configured on each node where they need to run.

The next step is to further configure the cluster resources from iManager. We have already configured the resources to mount the volumes and IP addresses in the beginning of this section. The only step we need to perform is adding the agent load and unload commands to the respective scripts.

You may remember that for NetWare it was a problem to stop individual post offices because all the agents would be loaded with the same GWPOA.NLM. Therefore, we used a separate address space to load the agents that enabled us to unload that specific address space when migrating a resource to another node or when bringing it offline.

For Linux we can just provide the name of the GroupWise resource we want to stop or start as a parameter for the grpwise load script. For example:

/etc/init.d/grpwise start gwdom1
/etc/init.d/grpwise start gwpo1.gwdom1

Notice that for the Post Office Agent the name must be specified as PostOffice.Domain.

Tip

If you are running the first version of GroupWise 7 without service pack 1, modify the grpwise load script to resolve an error. You will find several lines with a startproc command that look like the following one, but one line will not have the -f parameter. Add the -f parameter as in this example:

startproc -u root -f -t ${agent[4]} -p ${pid_file} ${agent[1]}
"@${agent[3]}" >& /dev/null

To configure the load scripts in iManager, select Cluster Options and modify the properties of each individual GroupWise resource. The load script must be like this example (for simplicity we left out the comments):

. /opt/novell/ncs/lib/ncsfuncs
exit_on_error mount -t reiserfs /dev/evms/lvm/cluster1/gwdom1 /grpwise/gwdom1
exit_on_error add_secondary_ipaddress 10.0.0.66
exit_on_error /etc/init.d/grpwise start gwdom1
exit 0

The unload script must be like this example (again, for simplicity we left out the comments):

. /opt/novell/ncs/lib/ncsfuncs
ignore_error /etc/init.d/grpwise stop gwdom1
sleep 8
ignore_error fuser -mk /grpwise/gwdom1
ignore_error del_secondary_ipaddress 10.0.0.66
exit_on_error umount /grpwise/gwdom1
exit 0

Note

Add the m parameter, as we did in the example, to the command where the agents will be killed when they did unload correctly (the fuser -k command).This will help identify the processes accessing the mount point.

The configuration of the software, the volumes, the IP addressing, and the cluster load and unload scripts are now finished for the first node. It is now time to test the new cluster resources for the domain and post office on this node. Because you have changed the load and unload scripts, you must first bring the resource offline and then bring the resource back online on the same node. Test the agents with a browser by accessing the HTTP port.

Note

After bringing the resources offline, check whether the volumes are really unmounted (mount command) and whether the secondary IP addresses are released (ip address). If not, first check the cause of this. You may have misconfigured one of the unload scripts, or a volume was in use; for example, if ConsoleOne was still accessing the domain database.

When the agents can be brought online successfully on the first node, it is time to test migrating the resources to the other cluster nodes where the GroupWise agents must run.

Before testing this, make sure that these steps have been performed:

  • The agent software is installed on each node.

  • The agents are configured on each node.

If this is all done, test the agent migration to the other nodes.

Post Configuration Tasks

The default location for the MTA and POA log files and other paths will be pointing to the local file systems of the individual cluster nodes. By default, the log files are stored in /var/log/novell/groupwise. It is a good idea to create a log directory under the domain and post office directories and modify the log paths from ConsoleOne. Figure 8.9 shows the configuration of the log file path for the MTA of our system. For the MTA configure the typical log settings and also the message log settings.

Figure 8.9. Change the log file paths to point to the shared GroupWise volume.

[View full size image]

Tip

If you will be enabling SSL for your GroupWise system, also configure the certificate key file path for the MTA and POA.

Adding a Second Post Office to the Existing Groupwise System on Linux

In the previous paragraphs we explained in detail how to set up a default GroupWise system. A minimum system contains one domain and one post office. We will now describe how to add a post office to an existing GroupWise system in a Linux cluster.

For this step we expect that you have already created the shared EVMS volume where the data for this post office will be created and also the cluster resource derived from the Generic_IP_Service template. Perform the following steps to add a second post office:

1.
Start ConsoleOne and log in to eDirectory. Select your GroupWise domain and from the GroupWise System Operations menu select Connect to connect to your GroupWise domain.

2.
In ConsoleOne select the container where the new post office object needs to be created. Select to create a new object and choose to create a GroupWise post office.

3.
In the Create Post Office dialog box add the post office name and provide the path to the shared volume where the post office database will be installed. In our example in Figure 8.10, this will be GWPO2 in the /grpwise/gwpo2 directory.

Figure 8.10. Configure the settings for the new post office.

[View full size image]

4.
In the next dialog box you must set up the link for the new post office. Just as with the first post office, this must be a TCP/IP link. The IP address, as we planned in Table 8.2, will be 10.0.0.68, and the port numbers will be as listed here:

  • Client/server port 1678

  • Message transfer port 7102

  • HTTP port 7182

5.
Click Finish to create the new post office.

6.
Start the GroupWise install script and configure the agent for this new post office. This will create a post office configuration file in the /opt/novell/groupwise/agents/share directory. Do not forget to disable the check box that would configure the agent to be launched at system startup.

7.
Copy the new post office configuration file to the other cluster nodes. You can use SCP, FTP, or any other networking utility for this. For example, you might use this command:

scp /opt/novell/groupwise/agents/share/gwpo2.poa admin@host:/opt/novell/groupwise/agents/share

8.
Configure the cluster resource load script and unload script. Use the same command syntax that was used for the primary domain and post office.

9.
Bring the resource offline and back online again to start the second post office in the system.


Clustering iFolder 3 on OES Linux

In this section we describe how to configure iFolder 3 on an OES Linux cluster. iFolder version 3 is not available for NetWare. It is based on Mono and that development environment is not available on NetWare. Novell does not have plans to port this software to NetWare. Because the software and online documentation for iFolder version 2.x on NetWare have been around for a while, we decided not to add that part to the book.

When installing the OES Linux servers that will become your cluster nodes, preferably do not configure iFolder 3 during initial installation. It is better to first configure the entire cluster, the shared storage for iFolder, and the cluster resources and then install the iFolder software on each node in the cluster.

Linux Cluster Preparations for iFolder

We start our configuration at the point when the cluster is fully functional and a shared EVMS volume exists to be used by the iFolder server. In our example the shared partition is mounted at the root of the file system in /ifolder. Do not forget to create this mount point on all nodes. By preparing the volume up front, we can easily activate the installation volume on each of the wanted cluster nodes. We start the configuration of iFolder with the resource online on the first node, as displayed in Figure 8.11.

Figure 8.11. We start with the volume and cluster resources configured.

[View full size image]

Note

There is no need to use NSS as a file system for iFolder. No rights are needed for end users to access the files directly. All operations are managed through the iFolder Apache server. So a traditional Linux file system is sufficient for this application.

Create a cluster resource based on the Generic_IP_Service template. Do not use the iFolder template because that is used for iFolder version 2.x. In the cluster load and unload scripts, configure only the mounting of the volume and the secondary IP address. The load script of the resource in Figure 8.11 is as follows (for simplicity we left out the comments):

. /opt/novell/ncs/lib/ncsfuncs
exit_on_error mount -t reiserfs /dev/evms/lvm/cluster1/ifolder /ifolder
exit_on_error add_secondary_ipaddress 10.0.0.70
#exit_on_error /etc/init.d/myservice start
exit 0

Notice that the command line where the service starts has been commented out. We use the resource only for setting up the disk and IP address.

What you need to plan is what secondary IP address will be used by the iFolder server. We will use address 10.0.0.70 in our example because our cluster servers are configured with 10.0.0.61 and 10.0.0.62 and the cluster master resource with 10.0.0.65.

Because iFolder works with a secure environment via HTTPS on the Apache web server, it is necessary to make sure that all nodes in the cluster use the same certificates for Secure Sockets Layer (SSL) communications. If you were to use the default server certificates, it would never point to the correct iFolder server when that service moves from node to node during its life cycle.

We therefore start with the configuration of these certificates. In a production environment where you have purchased a certificate from a trusted third party, such as VeriSign or Thawte, to be used for secure communications via the Internet, we suggest that you use that certificate. In the description that follows we create a certificate with OpenSSL that can be used for testing purposes. The certificates are as secure as those you buy, but they are just not verified and thus, for example, a browser would prompt the user with a message to say that the source of communications cannot be verified. Perform the following steps to configure your certificates:

1.
Prepare a DNS name for your iFolder clustered server in the network, and make sure that each cluster node is able to resolve that name. In our example we use ifolder as the hostname.

2.
On the host where the iFolder resource is online, open a terminal window and create a directory to store the certificate files on the shared volume; for example, with the command mkdir /ifolder/cert. Make this directory your current directory to ensure that all files are created in this directory and you do not need to provide full pathnames in your commands.

3.
On the same host create a private key in a file with the following command: openssl genrsa -out private.key 1024

4.
The generated private key file can now be used to request a certificate from the certificate authority (CA) on your host. For this we need to create a certificate signing request (CSR). Create such a request in a file with the following command:

openssl req -new -key private.key -out request.csr

Answer the questions with information that matches your environment. The most important answer is the Common Name to be used in the certificate. This must match the exact full qualified DNS name of your iFolder server. If your server, for example, can be reached at ifolder.OESCorp.com, that would be your Common Name.

5.
Now create the certificate that will be used by Apache with the following command. This will actually use the private key and the certificate signing request and have it signed by your CA.

openssl x509 -req -days 30 -in request.csr -signkey private.key
-out ifolder.cert

6.
Delete the certificate signing request rm request.csr.

7.
On every cluster node where the iFolder server will run, configure the Apache SSL configuration file to point to the newly created certificates. This file is typically called vhost-ssl.conf and is stored in the /etc/apache2/vhosts.d directory. Modify the two lines in the file on every node as in the two lines in our example:

SSLCertificateFile=/ifolder/cert/ifolder.cert
SSLCertificateKeyFile=/ifolder/cert/private.key

Tip

If your iFolder server will use a different hostname than other services that you host on the same server, you might want to consider configuring it as an Apache virtual host. For example, this might apply when there already is webmail.company.com for this server with an SSL certificate and now you want to use ifolder.company.com. In that case add the previous SSL configuration to the Virtual Host section. In most cases this issue will not be a problem because iFolder is an alias configured as /ifolder for the server's default URL. When setting up a virtual host, do not forget to copy the file inspection.wsil from /opt/novell/ifolder3/etc to the new document root directory, as described in the iFolder online documentation.

Installing iFolder 3 Software on All Nodes

At this point the iFolder 3 software and optionally the iFolder 3 WebAccess software must be installed on each node in the cluster where the software will run. There is no need to bring the cluster resource with the volume and IP address online on the other nodes for this installation. Perform the following steps to install the software:

1. | Start YaST and from the Software category select Install and Remove Software.
2. | From the Filter drop-down box choose Selections.
3. | In the Package Selection list select the iFolder 3 package and optionally the iFolder 3 Web Access package.
4. | Click Accept to start the software installation. Change CD-ROMs as needed to finish the installation.
5. | Go to the second and all other nodes and install the software.

Configuring iFolder 3 on the First Node

When the software is installed, it is time to configure iFolder. We start with the first cluster node where the iFolder server resource is currently online and where the IP address is also bound.

Perform the following steps to configure iFolder on the first node:

1.
Create the directory structure needed for iFolder on the shared volume. In our example the volume is mounted on /ifolder, so we create the directory with the following command:

mkdir -p /ifolder/.local/share/simias

(The -p parameter creates parent directories when needed.)

2.
Change ownership for all directories in the iFolder shared volume to the wwwrun user and www group:

chown -R wwwrun:www /ifolder

(The -R parameter changes the ownership recursively in all child directories.)

3.
Create a symbolic link for the newly created directory to the wwwrun user's home directory. Use the following command:

ln -s /ifolder/.local /var/lib/wwwrun/

The creation of the symbolic link can be tested with the following command (the link should be reported in the file listing):

ll /var/lib/wwwrun/

4.
Change ownership of the newly created symbolic link to the wwwrun user and www group with the following command:

chown -R wwwrun:www /var/lib/wwwrun/.local

The iFolder 3 software must further be configured from YaST. The full iFolder installation documentation is available from Novell's documentation website at www.novell.com/documentation. So we will only walk you through the most important steps without unnecessary details. Just follow these steps:

1.
From YaST, in the Network Services category select iFolder 3 to configure the server.

2.
Authenticate to eDirectory through LDAP with a valid username and password. For the LDAP server address enter the IP address or hostname of the iFolder server cluster resource.

Note

It is very important that you do not leave the default address of the local server in place when specifying the LDAP server. If that node fails, the other nodes would try to contact that server for LDAP authentication. By specifying the iFolder server itself, you will always have a functional LDAP server when the iFolder server is running.

3.
Enter a unique name for the iFolder server. In our example we will use iFolder1OESCorp as the system name. This name will appear in the end user's web page, if you use Web Access.

4.
For the system store path we will use a directory off the mount point of the shared volume that is currently mounted on this node, as displayed in Figure 8.12. In our example this is /ifolder/iFolder1OESCorp.

Figure 8.12. Use YaST to configure the iFolder server on the first cluster node.

[View full size image]

Note

Do not specify the root of the mount point, such as /ifolder, because the configuration script would try to delete it. It cannot because you cannot delete an actively mounted directory.

5.
Specify the username and password for the iFolder Admin user. Also enter a Proxy Context. In this context a proxy user will be created with rights to LDAP.

6.
When iFolder is configured, you will be prompted to restart Apache. Let YaST restart Apache for you.

At this point your iFolder server should be functioning correctly on this first node. Try to access your iFolder server by accessing it from a web browser. Use the following case-sensitive URL, with your own hostname or IP address, to access the server:

http://ifolder.OESCorp.com/simias10/Simias.asmx

If you are prompted to authenticate then the server is up and running.

7.
If you have also installed iFolder Web Access, configure that service now from YaST in the same way as the iFolder software. You will be prompted for an alias; this is the URL suffix that will be used for accessing the server. Leave this at the default of /ifolder on all nodes. Also leave the iFolder Server URL to be localhost, but change the protocol from HTTP to HTTPS: https://localhost.

If you have configured iFolder Web Access, test it by pointing your browser to the Web Access URL of your server with the URL suffix you specified during configuration, for example, https://ifolder.oescorp.com/ifolder.

Installing the Imanager Plug-Ins

To manage iFolder version 3, you must install the plug-ins for iManager. The iFolder management task that is included by default is for iFolder version 2 only. To install the module, log in to iManager and go to the configuration panel. (This is with the administrator sitting behind his desk.) Under the Module Installation role you will find an entry named Available Novell Plug-in Modules. If you select this option, the iFolder 3 plug-in will be listed and can be installed from here. When the plug-in is installed, restart Tomcat with the following command:

/etc/init.d/novell-tomcat4 restart

Allow Tomcat some time to restart and reload all its modules. If you try to access iManager too soon after reloading, error messages will appear. If you now log in to iManager, there will be a new Novell iFolder 3 task where you can manage your server.

Configuring the Apache Daemon

At this point the Apache server must be stopped and the iFolder cluster resources must be migrated to the second node where iFolder must be configured. Also configure Apache to no longer start at system startup. We want the cluster software to take care of that.

To stop Apache and prevent it from starting in the future at system boot, execute the following commands on the first node:

/etc/init.d/apache2 stop
chkconfig apache2 off

Configuring the Cluster Resource

With the software configured, the next step is to configure the load and unload scripts for the iFolder resource. In iManager under Cluster Options, mark the Resource and Select properties to configure the scripts. Here is an example of our load script (we left out the comments for simplicity):

. /opt/novell/ncs/lib/ncsfuncs
exit_on_error mount -t reiserfs /dev/evms/lvm/cluster1/ifolder /ifolder
exit_on_error add_secondary_ipaddress 10.0.0.70
exit_on_error /etc/init.d/apache2 start
exit 0

Here is an example of our unload script (we left out the comments for simplicity):

. /opt/novell/ncs/lib/ncsfuncs
ignore_error /etc/init.d/apache2 stop
ignore_error del_secondary_ipaddress 10.0.0.70
exit_on_error umount /ifolder
exit 0

When you are using the unload script in this way, it will stop the entire Apache server. This means that other services that use Apache will stop working. This could also include iManager for cluster management,\ because even while iManager software runs in Tomcat, the HTTP traffic is handled by Apache. In such cases it is possible to leave Apache running and just stop the Mono processes of iFolder. To configure this, remove the line from the unload script that stops the apache2 daemon and replace it with these two lines that will terminate the running mono processes:

[View full width]
ignore_error mono /usr/lib/mono/1.0/mod-mono-server.exe
 --filename /tmp/mod_mono_server_simias10 --terminate

ignore_error mono /usr/lib/mono/1.0/mod-mono-server.exe --filename /tmp
  /mod_mono_server_ifolder -terminate

This will stop the iFolder software and leave Apache running. It is simple to check whether the software actually unloads. When running the fuser command with the -m parameter, you can easily find out what process is accessing the iFolder shared volume mount point. For example, you might use this command:

fuser -m /ifolder

This will display the process id (PID) of the mono process. After executing the two command lines shown previously, the process will be terminated and the fuser command will show that no processes are using the file system at the specified mount. Thus it is safe to unmount it and migrate the volume to another node.

This also means that Apache should probably start when the server comes online. And you probably remember that we disabled this service earlier in this section. So when you configure this feature, also enable the service again with this command:

chkconfig apache2 on

When you are adding this feature to the unload script and you encounter problems with loading Mono again when the resource comes back online, add an Apache2 stop command in the load script just before the line where you mount the file system:

exit_on_error /etc/init.d/apache2 stop

This will ensure that any previously loaded modules are stopped and that all modules can be successfully loaded.

Testing the Cluster Resource

After you have configured the cluster resource scripts, it is a good test to bring the service offline and back online at least two times on the first node to see whether everything works as planned. If you would start migrating it to another node right away and there would be a problem, it might be harder to track down what the cause is.

Not only stop and start the service, but also access it through the web browser if you have Web Access installed or through the iFolder Client from a workstation. Those tests will show that iFolder works as planned.

If everything is okay, it is time to configure the remaining nodes in the cluster and test a migrate.

Configuring the Other Cluster Nodes

When the first node is fully functional and the resource can be brought online and offline successfully, it is time to configure the other nodes. Remember that we already installed the software from YaST on all nodes earlier.

You perform the same configuration for iFolder 3 and optionally iFolder Web Access from YaST as you did for the first node. Do not forget that, as on the first node, the LDAP server you authenticate to must be the iFolder cluster server.

When being prompted for a system store path, enter a bogus name that will be replaced later. All other configuration steps, including those of the Web Access Service, are identical to those for the first node.

When the configuration is finished, do not let YaST restart Apache. The software will come online when we migrate the resource to this node.

There are a few manual post-configuration steps you must perform:

1.
Copy the configuration file from the first node to all other nodes.

The file is named /opt/novell/ifolder3/etc/simias-server-bootstrap.config. It contains the configuration of the iFolder server that must be consistent through all cluster nodes.

2.
Create a symbolic link for the .local directory of the wwwrun user. This link will show up in red as invalid after you create it because the shared resource will not be available at this time. But as soon as the volume is migrated to this node, it will work fine. Use the same command as when you configured the first node, for example:

ln -s /ifolder/.local /var/lib/wwwrun/

3.
Change ownership for the newly created symbolic link to the wwwrun user and www group. Use the same command as when you configured the first node, for example:

chown -R wwwrun:www /ifolder

4.
Finally, depending on whether you want to keep Apache running independent from iFolder, disable loading Apache at boot time. When you have modified the unload script to stop only Mono and leave Apache running, it is not necessary to disable Apache with this command. In all other cases use the following command to stop Apache from loading when the server comes online:

chkconfig apache2 off

When the second node is configured, migrate the resource to that node and test iFolder. When the server is working correctly, configure the possible third and other nodes and test them individually.


Clustering the Squid Web Proxy on Linux

If you are using Open Enterprise Server Linux to run traditional Linux applications such as the Squid proxy server, Novell Cluster Services can be a good addition to increase the availability of those services. In this section you will learn how to set up the Squid server in an NCS environment. This is a description of a project we have realized at a large school in the Netherlands where the Squid proxy was used to provide access to the Internet for 12,000 students. Before realization of this project, Internet connectivity was often dropped because the proxy server was overloaded. Therefore, the customer wanted not only a high-availability solution in which there would always be a proxy to take over if one of the proxies failed, but also a load-balancing solution in which workload is divided among two operational proxies. Because load balancing is not something NCS can do, DNS round robin was used for this purpose. In order not to create a single point of failure, two DNS servers were installed: one as a master and the other as the slave.

DNS Round Robin Configuration

To handle the load balancing, the following entries were added to the DNS database:

squid    in    A     10.0.0.10
squid    in    A     10.0.0.20

The advantage of this solution is that DNS will automatically apply round robin; client connections will be evenly distributed among 10.0.0.10 and 10.0.0.20. The disadvantage of a solution like this is characterized as "black holing": If one of the nodes fails, the DNS server would never know and still forward clients to that server. In this scenario, however, packets are not forwarded to real servers, but to clusters where high availability has been taken care of. Therefore, this is no issue.

Squid High-Availability Design Considerations

For a server that works with a large number of configuration files that need to be the same on all cluster nodes, shared storage is an important issue. If a traditional Linux service such as Squid is clustered, it would make sense to use an ext3 or Reiser file system for shared storage. If, however, the customer does not care about the availability of the cache files because they can always be built up anew when a server fails, we can configure the cluster solution without shared storage for the proxy cache. Another advantage in this configuration is that no additional workload is created for the SAN, which is already under heavy use. For the purpose of this environment, it is not really necessary to create a shared storage device. Only configuration files need to be synchronized because the Squid configuration does not change often, and in this environment they can be better synchronized manually. As an alternative, rsync can be used to configure automatic synchronization of a limited number of configuration files. The design scheme for the solution described in this chapter is shown in Figure 8.13.

Figure 8.13. Design for a Squid high-availability solution.

[View full size image]

Creating the Squid Cluster Resource

Assuming that the NCS software is already installed and shared storage is available, the Squid cluster resource can be created. This involves a few general steps:

  • Install the Squid software on both servers in the cluster.

  • Prevent the Squid server from being loaded automatically.

  • Create a cluster resource for the Squid software.

The exact steps that have to be performed are described here:

1.
Start YaST, log in as root, and select the option Install and Remove Software. Then select the drop-down list in the upper-left corner of the YaST screen and select the Search option. Next, enter the search text Squid. From the results screen, just select Squid and click Accept to proceed installing the software.

2.
The next step is to prevent the Squid server from being started automatically when the server is booted; the reason for this step is that the cluster resource object will be responsible for starting Squid and not the Linux operating system itself. In YaST, select the runlevel editor from the System menu. Locate the Squid service and make sure that it is not selected.

3.
Start iManager at https://yourserver/nps and authenticate as the admin user to your tree. Select Cluster, Cluster Options and next use the magnifying glass to locate the cluster object to which you want to add the proxy server as a resource. Select this object; this will display an overview of all resources currently in the cluster.

4.
Click New under the Cluster Objects option to create a new cluster resource. You will now see a window where you can specify the type of object you want to create. Select Resource and then click Next to continue.

5.
Enter the name you want to use for the cluster resource. Select the generic IP service (see Figure 8.14) to inherit properties from and then click Next to continue.

Figure 8.14. Because there is no specific template for a Squid proxy server, you can use the generic IP service template to create the resource.

[View full size image]

6.
Now you can edit the sample load script to start the Squid service. The following load script will do fine:

#!/bin/bash
. /opt/novell/ncs/lib/ncsfuncs
exit_on_error add_secondary_ipaddress 10.0.0.31
exit_on_error /etc/init.d/squid start
exit 0

7.
In addition to from the load script, you will also need an unload script for this cluster resource that makes sure that the service is unloaded properly. The following unload script will do fine:

#!/bin/bash
. /opt/novell/ncs/lib/ncsfuncs
ignore_error /etc/init.d/squid stop
ignore_error del_secondaray_ipaddress 10.0.0.31
exit 0

8.
Next, you have to specify the failover and failback mode. You also need to specify all other parameters for your environment. For more details about these options, consult Chapter 6, "Cluster Management."


Clustering the Samba Server on OES Linux

The Samba server is rapidly becoming one of the most popular ways to share files in a Linux environment. Because the Samba server is available on OES Linux as well, we will discuss in this section how you can use it in combination with Novell Cluster Services. We recommend that you use Novell Cluster Services version 1.8 or later, which is included in OES, because it comes with a Samba resource template. The configuration of the Samba cluster resource involves the following steps:

1. | Create a file system on the shared disk.
2. | Create mount points to mount the shared file system on all servers.
3. | Install Samba on all servers.
4. | Configure the smb.conf file.
5. | Create the Samba clustered resource.

Creating the Shared File System

The first step in the creation of the cluster is to create a shared file system. This involves two steps. First, you need to set up the EVMS volume manager because this is the only volume manager that works in a clustered environment.

Next, you need to create at least one Reiser volume on that. There are two tools you can use to create EVMS volumes: the evmsgui utility and YaST2. Both can be started from the console of an OES Linux server. In this procedure, we will use YaST2 to create the shared storage. On this shared storage device, we will create a directory for the Samba configuration as well as all shares that are used in the Samba environment.

The following procedure describes how to create a shared storage device:

1.
From YaST, select System, Partitioner. Click Yes when a warning is displayed.

2.
Click Create to create a new partition. When asked on which disk you want to create the partition, select the shared storage device and click OK. Then indicate the partition type you want to create. It doesn't really matter what type you choose, but because in this example it will be the only partition on the shared storage device, primary will do fine.

3.
In the properties window of the partition you want to create, select Do Not Format. Then from the drop-down list, select the File System ID 0x8E Linux LVM. Then click OK to continue.

4.
Back in the main screen of the partitioner utility, click EVMS to start the EVMS volume creation program. Select the partition you have just created and then click Create Container to create an EVMS container object. You will always need an EVMS container if you want to create EVMS volumes.

5.
Select the device where you want to create the container. Leave the default container name set to system and then click Add Volume. This will add the device /dev/evms/lvm/system to your server. Next, click OK to close the Create EVMS Container screen. This will bring you back to the EVMS configurator main screen shown in Figure 8.15, where you will see the empty EVMS volume you have just created. From there, click Add to create an EVMS volume in that container. This will bring you to the Create Logical Volume screen.

Figure 8.15. The EVMS configurator shows you what EVMS volumes are created in the EVMS container.

[View full size image]

6.
Give the EVMS volume the name samba and specify the size you want to give it. Lastly, specify the directory where you want to mount this volume. /media/samba might be a good choice. In fact, you don't need to specify the mount point here because that is something you want to be handled by the NCS software, but YaST refuses to create the volume if you don't give it a mount point. The default file system used for this volume is Reiser, which is a good choice. Click OK to accept all settings. This brings you back to the EVMS Configurator main screen, where you can see that a device with the name /dev/evms/lvm/system/samba has been created. Now click Next and then click Apply twice to create the volume. Then quit the partitioner utility and close YaST.

You have just created an EVMS volume on one of the servers in the cluster. As you might expect, this will cause the EVMS volume to be active at that server all the times it boots. The other server, however, still does not know about this EVMS volume. To teach it that there is something like an EVMS volume on the shared disk device, you need to start the boot.evms script with the boot.evms start command. To make sure that it is activated each time the server restarts, use the insserv boot.evms command. (Or use the YaST runlevel editor to perform these tasks.) Then before continuing, check that the EVMS device file for the Samba volume you have just created is present on all other nodes in the cluster as well.

Creating Mount Points to Mount the Shared File System on All Servers

After creating the EVMS volume, you have to make mount points on it. This way you make sure that from the cluster resource load script, data can be written to the shared volume. Creation of the mount points and associated directories involves the following steps:

1. | Make sure that the mount point where the EVMS volume must be activated exists on all servers in your cluster. If on all servers in the cluster you want the shared volume to be mounted on /media/samba, make sure that the directory /media/samba exists. This is because the cluster load script will assume it is there when activating the shared volume. Create this directory manually with the mkdir /media/samba command on all servers. In the previous example, the mount point /mnt/samba was mentioned; use mkdir /mnt/samba on all servers to create it.
2. | On one server only, log in as root and mount the EVMS volume on the mount point you have just created. For example, use mount /dev/evms/lvm/system/samba /mnt/samba to mount the volume.
3. | On the EVMS volume, create all directories that are required. To store the configuration files, in /mnt/samba on the server where the EVMS volume is mounted, use mkdir -p etc/samba to create a subdirectory to store Samba configuration files. Also make sure that all shares that you want on the shared volume are created as directories as well, so that you can refer to them from the smb.conf configuration file.

Because it is important that the shared volume be loaded from the cluster resource load script and not from /etc/fstab, make sure that in /etc/fstab on the server where you have created the EVMS volume, all lines that refer to the volume are commented out. Of course, you can also remove the lines, but by commenting them out, at least you have one single place in which drive information is documented.

Installing Samba on All Servers

Next, you have to install Samba on all servers. Normally this shouldn't be a problem because it is installed in a default installation of OES Linux. If it is not, use YaST to install the software.

Configuring the smb.conf File

By default, the smb.conf configuration file is read from /etc/samba/smb.conf. In a clustered environment, you should make sure that it is read from the shared storage device. Therefore, first you have to move it to the shared storage device, and then you have to make some modifications to the contents of the smb.conf file that will be used by the cluster resource:

1.
Copy the /etc/samba/smb.conf file to the etc directory on your shared storage device. Then rename it to reflect the name of the Samba cluster resource. For example, use mv smb.conf cluster_smb.conf to rename it. For the remainder of this procedure, we will refer to the smb.conf file for the Samba clustered resource as cluster_smb.conf.

2.
Open the cluster_smb.conf file with an editor and add the following lines to the [global] section:

bind interfaces only = yes
interfaces = cluster resource ip address
pid directory = $SAMBA_ROOT/share/locks

Of these lines, the first line will make sure that Samba connects to only the interfaces specified with the interfaces parameter. The interfaces parameter should be set to the unique IP address you will assign to the Samba cluster resource. Last, the pid directory setting will make sure that the PID files that are created by the Samba server are written to a directory on the shared disk as well. In this parameter, the SAMBA_ROOT variable is used; you will set this variable later when configuring the Samba cluster resource.

3.
Next, also in cluster_smb.conf, modify the line netbios name = %h-W. Normally this line will use the name of your current host followed by -W as the NetBIOS name for the Samba server. Because the Samba server will be a clustered resource, this doesn't really make sense. Therefore, change it to something that will be unique and recognizable as your Samba clustered resource. A name like SAMBACLUSTER would be a decent choice.

Creating the Samba Clustered Resource

Now that all required modifications have been made to the smb.conf file, you still need to create the Samba cluster resource. Novell Cluster Services 1.8 comes with a template object to make creation of this resource easier. The following procedure describes how to create the Samba cluster resource:

1.
Make sure that the shared disk you have created for the Samba server is unmounted.

2.
Start iManager on https://yourserver/nps and log in as admin.

3.
In iManager, activate Clusters, Cluster Options.

4.
Specify the name of the cluster where you want to create the Samba resource, and then click New to create a new resource.

5.
From the New Resource window, select the resource type and click Next to continue.

6.
Use Samba as the cluster resource name, and in the Inherit from Template field, browse to the Samba_Template object. Make sure that the Define Additional Properties box is set and then click Next to continue.

Tip

If you don't see a Samba resource template, make sure that you are using OES SP1 or later. The Samba resource template exists only from this OES version on.

7.
You will now see the load script for the Samba resource that is created automatically based on the load script on the Samba resource template. Make sure that you adjust the load script to reflect your environment. Normally, the following parameters must be modified:

  • RESOURCE_IP: The IP address you want to assign to the Samba resource.

  • MOUNT_FS: The file system type you are using on the shared disk.

  • MOUNT_DEV: The name of the EVMS volume. If you have created the EVMS volume with YaST, the name will be something like /dev/evms/lvm/system/samba.

  • MOUNT_POINT: The name of the directory where you will mount the shared disk device. In this example we have used /mnt/samba for this directory name.

  • CONFIG_FILE: The name of the cluster_smb.conf configuration file you have created on the shared disk. Make sure that you include the complete pathname to this file as well.

8.
After clicking Next, you will enter the screen where you can configure the unload script. Make the same modifications as mentioned in step 7 of this procedure, and click Next when finished.

9.
Set the resource policies you want to use for this resource. Consult Chapter 6 for more information about these policies, and set the appropriate values. Then click Next to continue.

10.
Make sure that the right nodes are assigned. If this is the case, click Finish to finish the configuration of the Samba cluster resource.

You have now configured the clustered resource. Depending on what is currently loaded on your server, it may not yet be available to load. If currently the shared volume is mounted anywhere or the Samba server is loaded on any node in the cluster, the Samba cluster resource cannot load successfully. Therefore, first make sure that the Samba server is not currently active on any server by using ps -ef | grep smb. If this command shows any active Samba servers, first use rcsmb stop to stop the server. Then use insserv -r smb to prevent the server from being started automatically. Next, make sure that the shared device that will be used by the Samba server is not mounted anywhere. If it is, unmount it by using umount /media/samba (which will work only if the device is mounted on /media/samba). If you get a "device busy" message, use fuser /media/samba to display a list of all processes that currently have files open on the shared device. Next, try to terminate these processes the proper way. If you are having problems doing this, you can use fuser -k /media/samba, which will force the device to be unmounted. The latter method, however, should be used only as a last resort.

Making a Share Available from the Samba Clustered Resource

You have now created a cluster resource for the Samba server. The Samba server, however, is not sharing anything so far. Here we will discuss how to share a directory on the shared resource with the Samba server:

1.
On the shared disk device, use mkdir to create a directory. For example, use mkdir /media/samba/share to create a directory with the name share. Before issuing this command, use mount to make sure that the shared disk device is currently mounted.

2.
To access a share, two things are needed: rights at the shared directory at the Linux filesystem level, and permissions to access the share granted by the Samba server. To get access to the share at a Linux file-system level, you need to create an eDirectory group that can be used as a trustee at the Linux file-system level as well. This is called a LUM (Linux User Management) enabled group. You can enable an eDirectory group as a LUM group from iManager, Linux User Management. When the eDirectory group has been granted rights at the Linux file system, you can use chmod from the command line to grant rights to the group. If you would use, for example, a scenario in which some Linux users need to be able to write to the shared directory at the Linux file-system level, you would need to make a LUM group, add these users as members to this group, and next use chmod g+rw yourgroup to provide the necessary rights.

3.
As the next step, you have to configure the shared directory as a share from etc/cluster_smb.conf on your shared disk device. The entry to create this share would look like this example:

[share]
   comment = shared directory
   path = /mnt/samba/share
   read only = no
   inherit acls = yes

Of course, you should realize that this is a very basic example of how a share can be configured. For more details on the options you can use when configuring a Samba share, consult the man page of smb.conf.

4.
The Samba share on the shared disk device is ready for use now. You can start the Samba server and access the share from any location you want.

In this section you learned how to configure Samba as a resource in an NCS clustered environment. In fact, the Samba server can be used as a blueprint for any other Linux service that needs to write files to a shared storage medium. When you are creating such an application, be sure to make arrangements for the following:

  • Make sure that the configuration files of the service in question are on the shared medium.

  • Make sure that all data stored on the shared medium is referred to in the correct way by the service or its cluster resource load script.

  • Configure the cluster resource load script where the shared storage device is mounted on activation of the service. This will help you use your application in the clustered environment.


Summary

In this chapter you learned how to configure some important services in an NCS environment. Only some of the more common services have been dealt with: Two of them are typical applications from the Novell world, and two others are applications from the Linux world. Both these kinds of applications can be used in the high-availability environment offered by Novell Cluster Services. We have not been able to deal with all possible applications in this section. For information about other applications, consult the NCS documentation at www.novell.com/documentation or do a search in the TIDs at http://support.novell.com.


Chapter 9. Advanced Clustering Topics, Maintenance, and Troubleshooting

In this chapter you will find the answers to many of the situations in which the normal configuration options from this book will not work. When something is broken or not working as it should, the answer could lie in this chapter. We have also added some maintenance tips and information on advanced topics, such as working with certificates in a cluster environment, and cluster backups.


Advanced Configuration and Maintenance

Like every server environment, a cluster server environment is subject to change. Sometimes you initiate changes yourself and other times they come from a larger project. But in any case, there can be situations in which you have to perform maintenance on your cluster and change configuration options. This section will help you with that task.

Creating Cluster Documentation

For your NetWare or Linux servers you should already have common documentation in place. Without ever touching the server, you need to be able to find out from a paper source what the IP addressing scheme is, where in the network the servers are located, and even more detailed information, such as what the contents of your AUTOEXEC.NCF file are on NetWare.

For a cluster there is some extra documentation you should collect and add to your system documentation. The layout of cluster nodes, resources, resource priorities, and IP addresses used must be available not only from checking iManager.

Changing the Master IP Address

The cluster master resource is one that is always available on the master node in the cluster. It can be your main access for managing the cluster because whatever nodes fail, this resource and its IP address will always be available as long as at least one node in the cluster is still alive.

If you ever have to change the IP addressing of the cluster, this address must also be changed. You cannot change it in the master resource itself. You will have to change the address in the cluster general properties. Be aware that you will have to bring down the entire cluster to change this address.

To change the address, perform the following steps:

1. | In iManager select the Cluster Options task and select the cluster where you want to change the address.
2. | Click the Properties button under the Cluster Selection text box.
3. | In the General tab enter the new master IP address.

This IP address change will be in effect when the cluster is restarted. This requires that all resources and thus services will be unavailable. To bring the cluster down, issue the cluster down command on one of the nodes. When all nodes have left the cluster, restart the first node with the cluster join command and verify that it has loaded the master IP address. For this, use the display secondary ip address command on NetWare or the ip address command on Linux.

Changing the Heartbeat IP Address

If your cluster nodes have only one network adapter, the IP address of that single adapter is used in the cluster for the heartbeat operation. If you had performed the installation on a server with more than one adapter, the installation program would have asked which IP address to use. Whenever you want to change the address your cluster nodes are using to communicate for the heartbeat process, follow the procedure given next.

Note

In Chapter 3, "Clustering Design," we describe our idea about using, or rather not using, a dedicated heartbeat network. So if you are changing your addresses to set up such a network, we advise you to first read that section.

The cluster nodes must all be brought offline before implementing the new IP address scheme. Perform the following steps to change an IP address:

1. | Unload the Cluster Services software on all nodes.
2. | Configure the new IP addresses and, when necessary, the new network adapters on your servers.
3. | From ConsoleOne or iManager browse to the cluster node object of each individual cluster node and open the properties page.
4. | In the Cluster Node Identification tab enter the new IP address information for that node.
5. | When all nodes have been configured with the new IP address information, load the cluster software again on all nodes.
Renaming A Netware Server That Is An Iscsi Initiator

Renaming a NetWare server is always a cumbersome task. Of course, the procedure is simple: Change the name in AUTOEXEC.NCF and reboot the server. After the reboot, the eDirectory Limber process takes care of renaming the server object, and the change is synchronized to the other servers in the tree. But then the work begins because there will be login scripts, other scripts, batch files, and application objects with UNC paths that need to be updated. So, if possible, never rename a NetWare server.

If you ever have to rename your server and it is an iSCSI initiator, an extra step is needed because the server name is used in the server's registry for the name of the iSCSI initiator connection.

To change this name, you must edit the server's registry manually. To allow for modifications to be made through Remote Manager, unload PORTAL.NLM and reload it with the /regedit parameter:

LOAD PORTAL.NLM /REGEDIT

The example in Figure 9.1 shows the registry key that needs to be modified:

Figure 9.1. When renaming a NetWare server, also rename the iSCSI initiator name in the registry.

[View full size image]

My Server\Protocols\iSCSI\Initiator Control Parameters\Initiator Name

Modify the name here and reload the iSCSI initiator software with the IOFF and ION commands.


Troubleshooting Your Cluster Environment

First of all, an important remark: We can never be complete and up-to-date in a printed book. The most recent information for any troubleshooting issue can be found at Novell's support website at www.novell.com/support.

Tip

If you can spare a few bucks, it is also interesting to get a subscription to Novell's support knowledgebase on DVD. With a monthly update you get access to an integrated search engine on your own computer that will search through the TIDs, product documentation, application notes, and cool solutions.

Troubleshooting Resources

For NetWare and Linux there are different locations where you can find information about the workings of your cluster. But there is a common denominator. We will introduce the generic troubleshooting first and the platform specifics thereafter.

Cluster Event Information

Every event that occurs on your clusternot only errors, but also resource migrate actions, cluster membership changes, and other such normal operationswill be present in the Event log. The Event log is available from every management application that provides access to the cluster:

  • From iManager select the Cluster Event Log task from the Clusters role.

  • In ConsoleOne (NetWare only) the Event log is accessible from the Cluster State view. It is the second tab in this view.

  • In Remote Manager (NetWare only) the Event log is accessible from the link in the Cluster Management section.

Netware Informational Resources

NetWare does not keep log files of all cluster activity on the file system. Instead it writes most of the information that you will ever need to the Logger Screen. This screen is always the second screen on a NetWare 6.x server if you access the screen list with the Ctrl+Esc key sequence. You can browse the Logger Screen to find information about previous activity. And if needed, you can write the contents of the Logger Screen to a file called C:\NWSERVER\LOGGER.TXT by pressing F2. If you cannot find the information you need when browsing the logging screen, it is also possible to search through the contents of the Logger Screen with the LSS command from the server console.

Linux Informational Resources

Many Linux applications, including Novell Cluster Services, write information to the general log file /var/log/messages. Information about loading and unloading the software and errors in that process are stored in /var/opt/novell/ncs.

Re-creating the SBD Partition

When your cluster is installed with a shared Split Brain Detector (SBD) partition, the cluster nodes will automatically try to access this shared disk when the cluster software is loaded. When this partition cannot be found, a cluster node that tries to join will not finish trying to join the cluster. When this happens, it is of course a good solution to first find out why the SBD partition is not available. It could be something as simple as a fiber-optic cable not being attached, or the connection to an iSCSI target not being available. In those cases resolve that problem and reload the cluster software.

When the problem with the SBD partition is of a more serious nature, for example when the Logical Unit (LUN) where it existed was deleted, you should create a new SBD partition.

To create a new SBD partition, regardless of what platform you need to perform this action on, you will need empty space on a shared disk to do so. So either create a new LUN, or use a new iSCSI target or some free space on another type of shared disk for this. If there is a problem with an existing SBD partition and you need to re-create it, delete the partition from NSSMU or the YaST Partitioner and perform the steps given in the next subsections for the required platform.

Note

In a mixed-cluster environment you can perform the actions from either operating system. When you're performing this action on NetWare, it is necessary for the Linux nodes to be rebooted to detect the new partition being created. When you're performing the action on Linux, the NetWare nodes must be rebooted.

Netware SBD Installation

To create a new SBD partition on a NetWare node, you must first unload the Cluster Services software on all nodes with the ULDNCS command. This will also unload the SBD and SBDLIB NetWare Loadable Modules (NLMs) that need to be reloaded to re-create the SBD partition. Perform this task on each cluster node. This will effectively stop all services in the cluster.

Before creating the SBD partition, check that every node has access to the shared disk where the partition will need to be created. Use Remote Manager, MONITOR.NLM, or the list devices command for this task. Also use NSSMU or iManager to verify that a partition of type Cluster does not exist already. If it exists and you need to create a new one, remove the partition from NSSMU of iManager first.

On one of the nodes, perform the following steps to create the SBD partition:

1.
Load the necessary NLMs in this order: CLSTRLIB, SBDLIB, VLL, SBD.

2.
Load SBD.NLM. This will report that there is no SBD partition.

3.
Execute the SBD INSTALL command to start creating the partition.

4.
You will be prompted as to whether you want to create a mirrored SBD partition. If you have prepared two shared disk partitions for this, answer Yes. Otherwise, answer No.

5.
You will be prompted with a list of available devices to select the device where the partition needs to be created. (See Figure 9.2.) If you have selected to create a mirrored partition, a second prompt will ask for the device for that mirrored partition device.

Figure 9.2. When you're creating a new SBD partition, it can also be mirrored.

[View full size image]

6.
After the partition is created, you can start the cluster with the LDNCS command. The already-loaded NLMs will be skipped, the remaining NLMs will be loaded, and the node will join the cluster. Verify that the node is using the SBD partition with the SBD VIEW ALL command. When everything works as desired, load the clustering software on the other nodes.

Linux SBD Installation

To create a new SBD partition on a Linux node, you must first stop the Cluster Services software on all nodes with the rcnovell-ncs stop command. This will effectively stop all services in the cluster.

Before creating the SBD partition, check that every node has access to the shared disk where the partition will need to be created. Use the sfdisk -l command for this. Use the sbdutil-f command to verify that a cluster partition does not exist already. If it exists and you need to create a new one, remove the partition from YaST Partitioner first.

On one of the nodes, perform the following steps to create the SBD partition:

1.
Execute the following command, with the correct device specified as the parameter. In our example the partition will be created on the second SCSI device:

sbdutil -c -d /dev/sdb

To create a mirrored device, specify multiple devices with the -d parameter.

2.
After creating the partition, verify that it has been created with the sbdutil -f command.

3.
Next, restart the cluster software with the rcnovell-ncs start command.

4.
Verify that the cluster node is using the SBD partition with the sbdutil -v command. When everything works as desired, restart the cluster software on the other nodes.


Cluster Services Backup

In this section we describe possible backup solutions for your Novell Cluster Services environment. When designing the contents of this book, we decided to limit this section to a description of Novell's backup architecture (Storage Management Services [SMS]) and one third-party solution for backups, which has become Syncsort's Backup Express. The reason for narrowing the topic down to these two technologies is that there are too many possible solutions to discuss without this becoming a book on backups. We found, however, that we could not leave backup out of this book. We think that by describing these two technologies, we can give you enough information about how to use backup software in your cluster environment, regardless of the actual solution you select.

After a short introduction to cluster backups, we start with a description of Novell Storage Management Services, and after that we explain how to use Syncsort's Backup Express.

Cluster Backups Explained

When you have experience with backup software on NetWare or Linux, you might ask yourself why it is such a big deal to spend time on backups in a book about clustering. Your backups are as important in a standalone environment as they are in a cluster environment. There is, however, a very important difference when performing backups of cluster nodes.

When you configure a backup on a standalone server, you define a backup job that creates a backup of, for example, volume SYS and VOL1 on server FS1. When you create a backup in a clustered environment, you could also create a backup job to do that, but what if server FS1 has failed before or even during the backup and server FS2 now has volume VOL1 mounted?

In older backup applications the documentation for performing backups in a cluster explained how to create backup jobs that would try to create backups of all volumes on all available servers. Surely one of them would actually have the volume active and run the backup job. This is really like shooting a mouse with a cannon: You can't miss, but it is not really a refined method. And this type of backup does not survive a failover during the backup. Last but not least, every backup operation would by default show failures, so actual failures could easily be missed.

Newer backup applications, like the ones we discuss in this book, allow you to configure the cluster resources (pools and volumes) as individual entities that can be backed up independent of the server where they are active. And support for failover during a backup job is also supported.

Storage Management Services

Novell's implementation of this backup architecture has been around for many years and is now still available on NetWare and on Open Enterprise Server (OES) for NetWare and Linux. Storage Management Services is a backup framework containing standardized components that Novell's own backup application ties into and that third-party backup software vendors also use to back up data in a structured standardized way. Because of this setup with building blocks running on Open Enterprise Server for both NetWare and Linux, it is very easy for software developers to reuse the existing architecture and implement a solution. For example, when file-system data needs to be backed up, you only need to "talk" to the file system Target Service Agent (TSA), which provides access to the file system. When a new file system needs to be supported and a new TSA becomes available, the existing backup solutions can keep using the same modules for creating backups of the file system.

You can consider the SMS architecture as a tiered setup where the Storage Management Data Requester (SMDR) sits in the middle, as you can see in Figure 9.3. This software component plays a central role, providing third-party software developers with a means to communicate with the other components of the backup architecture and handling communications between servers.

Figure 9.3. The architecture of SMS is like a tiered setup with SMDR as the data broker.

Another important component of SMS is the Target Service Agent. This is the software that SMDR communicates with to collect the actual data from the servers. There are several implementations of TSAs available, for example, the file-system TSA (TSAFS) to back up files and TSANDS to back up eDirectory objects. The actual data (files or eDirectory objects) are called targets in the SMS scheme. Table 9.1 contains an overview of targets that exist in NetWare and OES environments and the TSAs that are available to create backups.

Table 9.1. Target Services and Their Corresponding Target Service Agents in OES

Target

Target Service Agent

Platforms

NSS file system

TSAFS

OES NetWare and Linux

Cluster resources

TSAFS

OES NetWare and Linux

Traditional NetWare file system

TSAFS

NetWare 6.x

Traditional NetWare file system

TSA500

NetWare 5.1

VFS-compliant file systems

TSAFS

OES Linux

NDS/eDirectory

TSANDS

NetWare

GroupWise

TSAFS

NetWare 6.x, OES NetWare and Linux

GroupWise

GWTSA

NetWare 5.1

With the TSAs listed in Table 9.1, all resources on our servers can be secured by our backup applications. Novell provides two backup applications, one for NetWare and one for Linux, to use the SMS architecture to create backups. These applications do not provide the full feature set that we need in a backup solution. For things such as tape rotation and keeping track of data in incremental and differential backups, a third-party application is needed.

Before we can use any backup utility, SMDR must be configured and functional.

An installation of Open Enterprise Server for NetWare and Linux contains the SMS software by default. You can check whether the software is installed from the GUI Install application on NetWare or from the Software Management entry in YaST. If the SMS software is not installed, this is also the location where to install it as a post-installation task.

Netware Configuration Files

The SMS software on NetWare is automatically started when the server boots because the load file SMSSTART.NCF is added to AUTOEXEC.NCF during installation. Any configuration data will be stored in smdr.cfg and tsa.cfg, located in the sys:\etc\sms directory.

Linux Configuration Files

On an Open Enterprise Server Linux installation, the SMDR daemon is configured to automatically load in runlevels 3 and 5. The daemon startup script is /etc/init.d/novell-smdrd. The actual configuration data is stored in smdrd.conf and tsafs.conf in the /etc/opt/novell/sms directory. When changes are made to these files, the SMDR and TSA can be restarted with the respective commands rcnovell-smdrd refresh and smsconfig -r tsafs.

Imanager SMS Configuration Options

On both NetWare and Linux, SMS can be configured from iManager. Configuration options for the SMDR are for protocol settingsfor example, which IP address the service will bind to, whether TCP/IP and/or IPX (NetWare only) will be used, and what mechanism will be used to discover services: SLP, SAP (NetWare only), or a HOSTS/DNS configuration. In a clustered environment SLP is the required discovery mechanism.

For the TSA most configuration options are for performance tuning, which, by the way, does not need any adjustments in most environments. The important setting for the cluster, as displayed in Figure 9.4, is that clustering must be enabled for the TSA when running in a cluster environment.

Figure 9.4. SMDR and TSAs are configured from iManager for both NetWare and Linux.

[View full size image]

Novell's Backup Software

These are the two applications Novell provides in Open Enterprise Server to create backups:

  • SBCON for OES NetWare

  • nbackup for OES Linux

The SBCON.NLM backup software for NetWare is a menu-based backup utility that runs on the server with local storage devices to create backups and restore data in a very simple way. We think that the menu, displayed in Figure 9.5, is self-explanatory. The only preparatory step you must take to run a backup is to have the correct TSA loaded. For example, load TSAFS.NLM or TSANDS.NLM.

Figure 9.5. On NetWare SBCON can be used to create backups or restore data.

[View full size image]

On Open Enterprise Server Linux the available backup application is nbackup. This is a command-line utility that can be compared to the tar command. It allows for creating a backup archive on a linear device or in a file. The difference between this and other archiving solutions is that it is SMS-compliant and works with the TSAs to create backups. It also writes to a newer file format, specific for tape backups: System Independent Data Format (SIDF).

A typical command to create a backup follows:

nbackup -cvf backup.sidf -U backupadmin -target-type=linux /data1

This command creates a new backup file named backup.sidf of all files under the /data1 directory. The command connects to the Linux file-system TSA on localhost as user backupadmin.

A complete reference of parameters that can be used with nbackup can be found in the manual page of nbackup with the following command:

man nbackup

Syncsort Backup Express

When selecting a third-party backup solution for our book, we investigated many products and vendors. Our selection was Syncsort's Backup Express for a number of reasons:

  • Syncsort is a well-known partner of Novell and works closely with it to develop its software. Backup Express therefore works with NetWare and Open Enterprise Server NetWare and Linux very well.

  • Backup Express is the only software that comes up in a search for the keyword "cluster" in the database of Novell's "Yes, Tested and Approved" program. You can check for yourself what the status is of this software and that of other vendors at the following URL: http://developer.novell.com/nss. Check for backup software under the category Backup/Restoration.

In this section we describe the architecture of Syncsort's Backup Express and how to get started with the software on Open Enterprise Server NetWare and Linux.

Syncsort Backup Express Feature Overview

Like every backup solution, this one performs the basic operation you need: to be able to store your data in a safe place in case something happens to the original data. But there are a lot of differences between backup software applications, in manageability, in supported hardware and software, or just in features. Here is a list of Backup Express features that we think are important for OES administrators:

  • Availability for UNIX/Linux, NetWare, and Windows. This gives you the flexibility to back up many platforms with one solution from one central location.

  • Support for clusters in Windows environments and for Open Enterprise Server Linux and NetWare. It also supports mixed-cluster environments.

  • Support for eDirectory and GroupWise databases through special database interfaces.

  • Support for NSS Snapshot technology. This enables you to make consistent backups in an environment with open files.

  • A storage area network (SAN) resource sharing option. This allows the software to attach tape devices to any server in the Backup Express environment.

  • Support for NDMP (Network Data Management Protocol) for backup to network attached storage (NAS) devices that support this protocol.

  • A centrally located catalog for enterprisewide backup control.

  • Browser-based management without the need to install a client application on a workstation.

For the purpose of our book, the most important feature is that Backup Express natively supports Open Enterprise Server clusters, not only in a pure NetWare or Linux environment but also in a mixed-node environment. This allows organizations that work with a current version of Backup Express to migrate easily to an OES Linux environment. Support for OES clusters is available from version 2.35 of Backup Express. This is also the version that the material in this book is based on.

Backup Express Architecture

In a Backup Express environment there are three server roles that need to be present:

  • Master server

  • Device server

  • Backup client

Figure 9.6 shows these roles in a typical Backup Express environment.

Figure 9.6. The necessary and possible components of a Backup Express implementation.

The master server can be either a Linux server or a Windows server. Its role is to initiate and control jobs. It also contains the catalog. The master server also runs the web server that can be used to manage Backup Express from a browser. This is the only way the backup system can be managed. The master server could be a cluster node, but we recommend that you use a separate server to play this role.

The device server takes care of writing to tape. This server does not have to be a dedicated server. It can also be an existing backup client. There can be more than one device server in a Backup Express environment. This allows for faster overall backups because data can be stored on more than one device at the same time. Tape devices can be attached to this device server via a SCSI connection, but they can also be attached to the storage area network through Fibre Channel. Backup Express also supports creating backups to a file system, which can be useful to test your backup solution or for creating backups of smaller clients to a central location.

The backup client is a server where the Backup Express CMAGENT software is loaded to provide access to back up the data. The backup client can be NetWare, Linux, and Windows.

Using Backup Express in a Cluster Environment

The configuration described in the previous paragraphs is not different for a cluster than it is for standalone servers. In a cluster the same serversmaster, device, and clientare needed. The only extra configuration step is to create virtual nodes that will be used to back up the cluster resources.

The use of virtual nodes allows Backup Express to keep track of resources to back up even when they fail over to another node during the backup. In the Backup Express management utility you define backups for local file systems, such as the SYS volume, for the physical servers. The backup of cluster resource file systems is defined at the virtual nodes. Figure 9.7 shows an example of a cluster with two physical nodes and three cluster volumes, for each of which a virtual node is created. The backup process of Backup Express runs through the standard Storage Management System SMDR and TSAs.

Figure 9.7. Backup Express uses virtual nodes to back up cluster resources.

[View full size image]

Getting Started With Backup Express on OES

If the cluster is up and running and your tape device is attached to either a server or the SAN, the Backup Express software can be installed.

On both OES NetWare and OES Linux the software is installed through the graphical interface. Before starting the installation, you must meet these requirements:

  • For the master server installation, make sure that you have the license key available for the product.

  • For the installation of other Backup Express servers, make sure that the master server is running.

Linux OES Installation

The first installation for a Backup Express environment is the installation of a master server. This can be either a Windows server or a Linux server. Not surprisingly, we explain how to install a master server on Linux in this section.

The installation must be started in a graphical environment. To start the installation, mount the installation CD-ROM on your server (this usually is done automatically), and run the install.bin executable. For example, this may be done with the following command:

/media/cdrom/Linux-oes/install.bin

This will launch the graphical installation program that will guide you through the installation procedure:

1. | After the installation is initiated, read and accept the license agreement to continue installation.
2. | Provide the path to where the software will be installed. The default path is /opt/backupexpress.
3. | Select the type of installation. This first installation will be a master installation.
4. | In the next screen select to start Backup Express automatically after a reboot. This will add an S95BackEx link to the /etc/init.d/rc5.d and /etc/init.d/rc3.d directories.
5. | When your master server is an Open Enterprise Server, the installation program will detect this and install the Backup Express OES modules.
6. | At this point you can add the installation server to the Backup Express environment. Perform this step only if your master server is also a node that you want to add to your backup environment.
7. | Provide passwords for your root user and your eDirectory administrative user account.
8. | Provide the key for your Backup Express software. You can either provide the key from a file or enter it manually.
9. | Select whether you want the system to accept tapes that are not yet labelled or that have a label from another system and automatically overwrite them. If you run only one system, we suggest that you always select Yes to make sure that your backup will always run. If you run multiple systems and someone could enter a wrong tape in your tape device, it is probably a better choice to select No to prevent the wrong tape from being overwritten.
10. | The last step of your installation is to provide the password for the sysadmin user that will be used to log in to the Backup Express management environment.
11. | An installation summary is displayed with an overview of your installation choices. Review them, and when everything is okay, select Install to start the installation.
12. | When the installation has finished successfully, the software will automatically be started.

With the master server running, you can now add nodes to back up to the environment or log in to the management interface. Adding a Linux node requires that you perform the same steps as described before for the master server, except that you select to install a client.

Netware Node Installation

The way the installation program for Backup Express can be started is different for the available versions of NetWare.

For NetWare 5.1 and 6 the installation is started from the graphical screen. If that is not running, execute the command startx to start the GUI. To perform the installation, follow these steps:

1. | From the Novell menu select Install.
2. | In the installation screen select Add.
3. | In the installation path field provide the path to the installation CD-ROM to start the installation process.

For NetWare 6.5 and Open Enterprise Server the installation can be started from the installation CD-ROM with the following command (where cd2350nw is the CD-ROM volume name):

java -jar cd2350nw:\netwareOES\install.jar

This will start the graphical installation application that will guide you through the steps of installing a NetWare server as a client to back up:

1. | Read and select the license agreement.
2. | Provide the path to the installation directory. The default is SYS:\BACKEX.
3. | In the last screen select Install to start the installation.
4. | Select to install the server to your Backup Express environment. If you select not to do this, you can add it later.
5. | If you selected to install the node to the backup environment, provide the hostname or IP address of the master server.
6. | The software will now be installed and started on the NetWare server. The installation will add the load command BEXNW to the server's AUTOEXEC.NCF.
Managing Backup Express

After the installation is complete, you can log in to the management console through an Internet Explorer browser. Other browsers are not supported with the current version (2.35) of Backup Express.

Open a browser window and enter the following URL:

http://hostname_or_ipaddress:6122

The Backup Express management software requires a running Java Virtual Machine (JVM) on your workstation. If the Java environment is not available, you can download the Microsoft JVM from the Backup Express server, where it is installed during installation of the master server. Both a URL to this download and one to download the JVM from Sun are available from the first screen after the management web page is loaded.

When the Java environment is running, a login dialog box will be displayed. Provide the sysadmin username and the password you specified during installation. The default password for the system is sysadmin. Figure 9.8 shows a default configuration of a cluster with both NetWare and Linux in the web-based administration utility.

Figure 9.8. During a migration, Backup Express can back up both NetWare and Linux.

[View full size image]


Chapter 10. Upgrading Clusters

In this chapter we describe how to upgrade existing clusters to newer versions of the Novell Cluster Services software. Both NetWare and Linux are covered in this chapter, as well as upgrading via the mixed-node cluster route. These are the upgrade scenarios that are covered in this chapter:

  • Upgrading a NetWare 5.1 cluster to NetWare 6.5

  • Upgrading a NetWare 6 cluster to NetWare 6.5

  • Upgrading a NetWare 6 or 6.5 cluster to Open Enterprise Server Linux

  • Migrating to a new cluster, for all versions of NetWare

Note

The code base of NetWare 6.5 with Support Pack 4a is effectively the same as Open Enterprise Server with Support Pack 1. Also, NetWare 6.5 with Support Pack 5 is identical to Open Enterprise Server (OES) with Support Pack 2. Therefore, where we talk about NetWare 6.5 you can also read "Open Enterprise Server NetWare."

Tip

The method of migrating to a new cluster can also be used to move existing servers into a cluster if the existing servers cannot be added as regular nodes, or when servers will be consolidated.

We start this chapter by describing the available upgrade scenarios from a functional and practical point of view before diving into the technical details of each solution.


Upgrade Methods Explained

In general there are three upgrade methods available for upgrading servers, and these are also the methods for upgrading clusters:

  • Upgrading the existing hardware and software, also called an in-place upgrade

  • Upgrading to new hardware and software in the same environment, also called an across-the-wire migration

  • Installing completely new hardware and software from scratch

These three upgrade methods and their pros and cons are discussed in the text that follows. Every environment is different and many parameters are important to making a decision for your specific approach to upgrading, such as available budget, technical state of your existing environment, administrator knowledge level, and the knowledge and resources in your upgrade project team. We have tried to give you the guidelines to find your own best approach.

Upgrading Existing Hardware and Software

If this approach is technically possible, it is the fastest and simplest method to perform an upgrade. Just load the CD-ROM into the server, run the upgrade, and you are ready. But unfortunately this method is not always supported. Besides any technical limitations, it is also the most risky method, and it might not be possible if your current hardware and software are too old.

Let us start with the practical pros and cons before discussing the technical ramifications. The most important benefit is ease of installation and the short amount of time needed. An upgrade from a NetWare 6 cluster, for example, to a NetWare 6.5 cluster can be performed by starting the server upgrade from the NetWare 6 server's graphical installation screen the same way as you would normally install a new product. The upgrade can be started while users are working, up to the point at which a reboot is required. This can be performed on all cluster nodes. When the first server is ready, it can be restarted, after which the upgrade will be completed. Per server, this process takes only a very short time. No data has to be copied via the network, which is always time-consuming. With some extra checking and maybe some manual tasks for upgrading services, such as backup software, the upgrade takes two to three hours per server, and some tasks can be performed simultaneously on multiple servers.

The largest disadvantage of this in-place upgrade method is that the risk is the highest. If during the upgrade a problem occurs that prevents you from moving forward, the path back to the original situation is the most complex and time-consuming. Because the software upgrade was written to your disks, it could even require a restore of your original server. Depending on the type of upgrade you are performing, the process might even have upgraded the storage area, such as when upgrading NetWare 5.1 with Novell Storage Services (NSS) version 2 to NetWare 6.5. In such a case a restore could even mean restoring all your data if you decide to roll back to your starting point.

When you want to perform such an upgrade, it can be a good idea to perform the upgrade in a test environment first. And, of course, create a backup before you start the upgrade procedure.

Tip

Setting up a test environment can be very time-consuming. But with the use of imaging software, such as Storage Manager from Portlock Software, setting up a test environment is nothing more than copying servers from the production machine to a test machine. For a cluster you will need to copy all nodes, of course. Using VMware virtualization software with a Physical to Virtual migration method in this process can save you a lot of hardware resources, all depending on the amount of data you need to copy to your test environment and the number of nodes you need to run in the test environment. If sufficient resources are available, you can also rent the hardware needed to set up the test environment.

Something very important to check and test if you are going to upgrade existing hardware to be used with a newer operating system is support for network and storage adapters and other hardware used in your environment. Make sure that drivers are available and that you have them on hand when you start the upgrade.

Besides these practical pros and cons, there are also technical issues to keep in mind. First of all, an upgrade might not be supported at all. A NetWare server cannot be upgraded to an Open Enterprise Server running Linux with an in-place upgrade. Other technical limitations can be the available hardware resources in the server to be upgraded. Whereas a NetWare 5.1 server could run with 512MB of RAM without too many problems, for NetWare 6.5 this really is the minimum and rarely enough. Also, disk space can be a problem. You should check for simple things before upgrading. A server with a 100MB DOS partition just cannot be upgraded to NetWare 6.5 without first making more disk space available. The NetWare 6.5 preupgrade check, available from Deployment Manager, can help you with these preparatory checks.

Migrating to New Hardware and Software

With this approach to upgrading to a new cluster environment, the original servers remain intact for the largest part of the migration process. The new servers are installed into the network with the new operating system and software, and can be thoroughly tested. After the new servers are found to be okay, the migration process can be started to copy the data to the new environment and replace the servers in the eDirectory tree with the new ones.

This procedure is safer than an in-place upgrade process because for a large part of the procedure, the original servers remain intact. If something goes wrong, you can easily revert to the original scenario. Only when you have reached a certain point does it become harder to revert to the original scenario. When you're using the NetWare Migration Wizard, this is when the source and destination server's identities are swapped. When a problem occurs in that step, it can be a very complex process to restore to the original situation. But even in that scenario the data will always still be available on the old server. When you're using the Server Consolidation Utility or when you're performing a manual migration with your own procedures, there will still be a point where you have to switch to the new servers. But if you have performed the migration yourself, you will probably be able to return to the original environment much easier.

Migrating From Netware to Netware

In a clustered environment using the NetWare Migration Wizard is not impossible, but we suggest not to use it for upgrading to NetWare 6.5. If an in-place upgrade is not possible and you want to build new servers and do not want to add them to the existing cluster or even tree, we suggest that you use the Server Consolidation Utility or perform the migration manually. If you are also setting up a new SAN, copying the data will be the process that is the most time-consuming. For other tasks that need to be performed, when you want to perform the migration yourself without the available utilities from Novell, there are tools available, such as trUSTEE.NLM, that can be used to create a list of trustee assignments on the source server and deploy them on the destination server. For Linux there is an equivalent tool available called metamig, available on OES Linux servers in the /opt/novell/nss/sbin directory. Other tasks that need to be performed include pointing users to the new cluster with their home directories. For that there are also many tools available, such as Homes from www.hbware.com. But even that step can be performed with tools you already have, when you know how to use them, for example with an LDIF export and import to modify the attributes.

Migrating From Netware to Linux

When your new cluster is an Open Enterprise Server Linux cluster, we suggest that you take the mixed-cluster route to migrate your servers. How to do that is explained later in this chapter, in the section "Upgrading a NetWare 6.5 Cluster to Open Enterprise Server Linux." The main advantage of this method is that all your data on your NSS volumes remains intact, and trustee assignments and server references, including home directories, stay in place if they were originally pointing to the cluster.

This will work only when your current environment is a NetWare 6.5 cluster. If you have an older environment with NetWare 5.1 or later, data and trustees can be transferred to OES Linux using the Server Consolidation Utility, which is part of the Server Consolidation and Migration Toolkit.

Migrating From Windows to Netware or Linux

In the event that you have a Windows environment that you want to migrate to an Open Enterprise Server environment with either NetWare or Linux, you can use the Server Consolidation Utility to migrate your Windows server to Open Enterprise Server NetWare or Linux. The destination server can be a standalone server or a cluster node.

Installing New Cluster Hardware and Software from Scratch

In the previous two topics we have described how to upgrade or migrate in the existing environment. This means that you keep your existing eDirectory tree with everything in it, including trustee assignments and other specific information in your storage environment. We have worked at customer sites where the decision was made to build a new environment alongside the existing one, complete with a new eDirectory tree, and to move to this new environment by pulling the plug of the old servers and having users log in to the new environment.

Such an upgrade path brings a lot of manual labor, and you will need much expertise. User objects and other eDirectory objects, such as application objects, must be re-created, and in many cases this requires exporting existing information and importing it into the new environment.

The reason to start with such a project is very often that customers see problems in their environment that they expect to remain there after the upgrade. For example, there may be application objects that contain a snapshot of a Windows environment that distributes unwanted settings to destination computers.

It is our belief that in all situations it is possible to migrate to a new environment with the existing eDirectory tree using the utilities supported by Novell such as the Server Consolidation Utility. It only requires you to have the right expertise in the project team. For example, you would need an engineer with eDirectory knowledge to analyze the existing environment, perform health checks, and prepare the environment for the new operating systems. And maybe a ZENworks specialist who prepares applications to be used in the new environment by moving snapshot applications to MSI-based distributions.

The bottom line here is that customers are sometimes afraid to migrate garbage into more garbage. But when cleaning up during the process is done by experienced garbage men, the new environment can run bright and shiny.


Technical Upgrade Methods Explained

In this section we explain the technical procedures needed to upgrade clusters for NetWareto-NetWare upgrades and for NetWare-to-Linux upgrades. In the last part of this section, you learn how you can also use the Novell Consolidation and Migration Toolkit to migrate NetWare and even Windows servers to NetWare and Open Enterprise Server Linux.

Upgrading a NetWare 5.1 Cluster to NetWare 6.5

This in-place upgrade is the only one from this chapter in which the service to your end users will be interrupted for a longer time. Clusters that run with NetWare 5.1 servers that need to be upgraded to NetWare 6.5 cannot stay online while being upgraded. A direct upgrade to OES Linux is not supported. If you want to take that giant leap forward then NetWare 6.5 is a required intermediate step.

A NetWare 5.1 cluster must be prepared with Deployment Manager. The reason for this preparatory step lies mostly in the fact that the upgrade will also be an upgrade from NSS version 2 to version 3. File-system trustee rights are handled differently in the new version used since NetWare 6. The local NDS database object ID for objects is no longer used; it is replaced by the object's Global Unique Identifier (GUID).

Note

Because file-system rights were stored in the file system with local object identifiers from the NDS/eDirectory database, the rights had to be modified when the volume was migrated to another server. The trustmig process took care of that. Because the rights are now stored by using the GUID, which is not server-centric, the trustees are the same on every server and the trustmig process has become obsolete.

Meet The Netware 6.5 System Requirements

Before upgrading the cluster, you must check whether you meet the system requirements for your upgrade to NetWare 6.5. Check the items in the following list to determine whether you are ready for the upgrade:

  • Check the hardware requirements of the server to see whether you have sufficient storage space and memory available. Also verify that your hardware is supported for NetWare 6.5check whether drivers are available for all devices and so forth.

  • Verify that all applications you run on the servers are supported on NetWare 6.5. If they need to be upgraded, uncomment them in the cluster load script before you start the upgrade. After the cluster upgrade, when you have upgraded the applications to the new version, uncomment them again. Do not forget to check your backup software.

  • Make sure that you obtain licenses for all nodes. NetWare 6.5 comes with two nodes, so when upgrading a cluster with three nodes or more, make sure that you have the extra licenses at hand.

Prepare the Netware 5.1 Cluster

To prepare the cluster for an upgrade, perform the following steps. Be aware that the cluster will be stopped by this process. So it really is a preparatory step you take right before upgrading the servers to NetWare 6.5.

Note

It is important to use the Server Health Utility from Deployment Manager before even considering upgrading your cluster. You first want to verify whether the servers can successfully be upgraded before you perform these steps. You do not want to find out halfway through the server upgrade that not enough disk space is available, for example.

Here's what you should do:

1. | Log in to the NetWare 5.1 cluster.
2. | Start Deployment Manager (NWDEPLOY.EXE) from the root of the NetWare 6.5 operating system CD-ROM.
3. | From Deployment Manager, under the first menu named Network Preparation, select the option Prepare a Cluster for an Upgrade.
4. | Click the link Prepare a Cluster. This will start the cluster installation program.
5. | When prompted, enter the required information: name of the cluster object, tree name, and context where the cluster object is located. Click Next to continue the upgrade. All cluster nodes will now be contacted by the Cluster Services Installation application. You can watch their loading from the server console on each node.
6. | When the process is finished, a message will be displayed that Novell Cluster Services has successfully pre-upgraded. The command to load cluster services (LDNCS.NCF) will now be commented out in the server startup file on every node, and a command to not activate NSS volumes will be added (NSS /AutoDeactivateVolume=ALL). These commands will need to stay there until after the servers are upgraded and will be updated automatically after the cluster upgrade.
Upgrade the Netware 5.1 Servers

At this point you must upgrade the cluster nodes to NetWare 6.5. Follow the directions from Novell's documentation website at www.novell.com/documentation.

Tip

You will not be able to mount the NetWare 6.5 operating system CD-ROM as a volume because this NSS feature has been disabled because of the preparatory step you have just taken. (Traditional volumes, so also SYS, will remain operational.) You could start a remote upgrade, but you can also copy the contents of the OS and product CD-ROMs to a subdirectory on your SYS volume. From the GUI installation you can point to that location. A local upgrade is faster than a remote upgrade.

From the server's graphical screen select Install from the main menu to start the upgrade. A list of currently installed products will be displayed. Select Add and point to the installation files to start the upgrade.

Alternatively, the upgrade can be started with the server up and running from Deployment Manager on a workstation. In the menu Install/Upgrade Options select Upgrade to NetWare 6.5. Select Upgrade a Server Remotely to start the server upgrade. This remote upgrade will be slower than an upgrade started from the server's graphical environment.

Upgrade the Cluster Software and Configuration

After all the servers are upgraded to NetWare 6.5 and are up and running, you can continue with upgrading the Cluster Services software:

1. | From Deployment Manager in the Post-Install Tasks menu, select the option Install/Upgrade Cluster. Click the link Install or Upgrade a Cluster to start the Cluster Services installation program.
2. | In the Cluster Services Installation Wizard select Upgrade Software in Existing Cluster and click Next.
3. | When prompted, enter the required information: name of the cluster object, tree name, and context where the cluster object is located. Click Next to continue the upgrade. All cluster nodes will now be contacted by the Cluster Services installation application. You can watch their loading from the server console on each node.
4. | Provide a master IP address for the cluster. This is a new feature compared to the clustering version of NetWare 5.1. This IP address must be an available address in the same subnet as the cluster nodes and the existing cluster resources.
5. | When prompted to automatically start the cluster software after the upgrade, select Yes or No. If you select No, you must manually start the cluster software on each node with the LDNCS command before going to the next step.
6. | Start ConsoleOne to upgrade the cluster configuration. Browse to the cluster container object of the cluster you have just upgraded.
7. | Right-click the object and from the Views menu select Cluster State View.
8. | You will now be prompted as to whether you want to complete the cluster upgrade. This process takes only one or maybe just several seconds. It updates the cluster volume resources that were updated before the upgrade with several temporary commands that were needed to perform the volume upgrades and restore the trustees. These commands will be deleted by this process. Also, the NSS pools in eDirectory will be renamed to reflect the name of the new cluster servers. NSS pools are new to NetWare 6.x and did not exist in NetWare 5.1.

The cluster upgrade is now complete. Post-upgrade tasks include upgrading any applications that require an update to run on NetWare 6.5, including your backup software.

Upgrading a NetWare 6 Cluster to NetWare 6.5

For an in-place upgrade of a NetWare 6 cluster to NetWare 6.5, two methods are available: upgrading the cluster while it is not available to end users and upgrading while the cluster remains available. Which upgrade method you choose is up to you. If you have the time to bring down the cluster and upgrade the servers during off-hours, that will probably be your choice. In many cases maintenance windows have become smaller, and for administrators who have a family life, the live upgrade scenario, also called rolling upgrade, will be very welcome.

Meet the Netware 6.5 System Requirements

Before upgrading the cluster, you must check whether you meet the system requirements for your upgrade to NetWare 6.5. Check the items in the following list to determine whether you are ready for the upgrade:

  • Check the hardware requirements of the server to see whether you have sufficient storage space and memory available. Also verify that your hardware is supported for NetWare 6.5check whether drivers arevailable for all devices and so forth.

  • Verify that all applications you run on the servers are supported on NetWare 6.5. Do not forget to check your backup software.

  • Make sure that you obtain licenses for all nodes. NetWare 6.5 comes with two nodes, so when upgrading a cluster with three nodes or more, make sure that you have the extra licenses at hand.

Upgrade the Netware 6 Servers With a Downed Cluster

This in-place upgrade requires only one preparatory step: installing the cluster licenses. If you are running a two-cluster node, no licenses are needed and the servers can be upgraded. With three nodes or more, first install the cluster licenses from iManager.

When the licenses are installed in the tree, upgrade the servers individually to NetWare 6.5. Before starting the upgrade of the first server, stop the cluster with the cluster down command. This prevents a scenario in which a server would come online with the new software while the old cluster is still running. That is part of the rolling cluster upgrade scenario and not what we started here.

Tip

It is a good idea to disable the command that starts the cluster (LDNCS.NCF) before upgrading. This helps in the way that you can first focus on a successful upgrade of all your servers to NetWare 6.5. Check whether everything runs as it should and that all drivers work okay, and then load NCS to check whether your cluster comes online successfully. Do not forget to enable the LDNCS.NCF load command again.

After the first server is upgraded and brought online, it will start the cluster services software. There are no updates that need to be performed to the file system or cluster resources such as with the NetWare 5.1 upgrade, so the cluster will be available immediately after the upgrade.

Perform a Rolling Upgrade of the Netware 6 Cluster

With this in-place upgrade scenario, the servers will be upgraded one at a time, leaving the cluster up and running during the upgrade. Services to the end user will be interrupted for a minimal amount of time. Effectively the cluster resources will be brought offline and online, which has the same effect as migrating a resource from node to another node. So it depends on your applications whether the users will notice the upgrade at all.

Tip

You can also perform the entire upgrade operation during normal operating hours and then perform only the last step during off-hours.

Before the upgrade is started, the cluster licenses for the NetWare 6.5 cluster must be installed. If you are running a two-cluster node, no licenses are needed and the servers can be upgraded. With three nodes or more, first install the cluster licenses from iManager.

When the licenses are installed in the tree, perform the upgrade of your servers through the following steps:

1. | Start the upgrade of the first server you want to upgrade to NetWare 6.5. Migrate any resources that are running on this server to other nodes. You can let the process do this automatically for you, but you might want to choose manually where to run these resources. Also, this guarantees that it happens when you want it to happen.
2. | The upgrade can now be started as a normal upgrade, described in the NetWare 6.5 online documentation at www.novell.com/documentation.
3. | After the upgrade of the server is finished, let it reboot (twice) until the upgrade is finished. The Cluster Services software will be started automatically after the second reboot.
4. | Resources can now be migrated back to this server. At this time you will have a cluster that contains nodes running Novell Cluster Services version 1.6 (form NetWare 6) and version 1.7 or 1.8. Up to NetWare 6.5 sp3 version 1.7 was used, and for NetWare 6.5 sp4 and Open Enterprise Server version 1.8 was used. This mixed cluster runs without limitations, but of course it should not be a permanent setup. Keep the cluster static in terms of configuration. It is also not possible to reconfigure the storage during the upgrade procedure.
5. | Continue with the upgrade of all other nodes in the cluster by performing the steps through step 4.

After the upgrade of the last server, the cluster that is running as the master node will finish the upgrade process of the shared disk resources with the cluster upgrade command. You can watch this process start at the server's logger screen. A message will be displayed from the Media Manager regarding upgrading the volumes (MM_Upgrade). When the cluster upgrade has finished, a cyan blue message will be displayed: Operational Mode 0. The upgrade will also be listed in the cluster Event log. From the online documentation, this is the description of what the upgrade process performs:

1. | All cluster resources on shared pools are temporarily brought offline.
2. | Media Manager software is called to upgrade disk media on one server. Disk media is then refreshed on all remaining servers in the cluster.
3. | All cluster resources on shared storage are moved back to their original nodes.
4. | The load scripts for each cluster resource run and each cluster resource is brought online.

The cluster upgrade is now complete. Post-upgrade tasks include upgrading any applications that require an update to run on NetWare 6.5, including your backup software.

Upgrading a NetWare 6.5 Cluster to Open Enterprise Server Linux

The upgrade procedure described in this section will always be an upgrade to new servers and not an in-place upgrade. The difference from the first upgrade described in this chapter, migrating to a new cluster, is that in the scenario described here the cluster remains intact, the Linux nodes are attached to the same shared storage, and in most cases service to your end users will not be interrupted. The migration scenario requires you to build a new cluster and move your data and applications to the new cluster. The migration scenario is supported for all versions of NetWare and even Windows and can also be used to build a cluster with the data and applications from a standalone server. The upgrade described here is supported only for clusters running NetWare 6.5.

Tip

Starting with Open Enterprise Server Linux Support Pack 2, this procedure also works for NetWare 6.0 clusters.

The upgrade path that we will walk you through in this section is that of extending the NetWare cluster with Linux nodes, which creates a mixed cluster. We will then later remove the NetWare nodes, leaving you with a Linux-only cluster.

Things You Should Know About a Mixed-Cluster Environment

When you start with a NetWare cluster and add a Linux server, we call that cluster a mixed cluster. Such a cluster runs fine. The two operating systems can coexist, and cluster resources can be migrated from NetWare to Linux and vice versa. That sounds like a great solution for environments where some applications would perform better on NetWare and others on Linux. It gives you the flexibility to have some applications available on the entire cluster, such as user data, while other applications run only on specific NetWare or Linux nodes. Such a cluster, however, has some limitations that we explain here. After you've read them, we hope you agree with us that running a mixed cluster is not a permanent solution. These are the limitations:

  • After you've added a Linux node to a NetWare cluster, it is no longer possible to add new NetWare nodes. Only Linux nodes can be added.

  • In a mixed environment you cannot add storage to the cluster or modify the existing storage pools. There is a workaround to create storage on NetWare with the Linux servers turned off in which they will identify new storage after the upgrade, but that really is not practical for normal operation.

  • File-system trustee assignments are calculated for NSS access via NCP the first time a volume is migrated to Linux. On Linux trustees are stored in an XML file on the volume. When rights are modified on NetWare, the Linux server will not automatically detect these changes in its XML file and vice versa. (There is a command for that.) So moving storage around from operating system to operating system is not something to work with in a highly active productive environment.

  • Not all applications can seamlessly migrate from NetWare to Linux. They need to be modified to run on Linux and must stay there after the conversion. Examples are GroupWise and iFolder 2.

  • Cluster resource scripts will automatically be translated from NetWare commands to Linux commands. But if a resource is created on a Linux node in a mixed cluster, the resource will not run on a NetWare node.

From the preceding information you can decide for yourself whether you want to work with this. But we strongly advise you to work with this mixed-cluster scenario only for the time it takes to upgrade to Linux. During the time the cluster is in a mixed mode, we suggest that you keep the configuration static. It should be a period during which no new applications will be added to the cluster and a clear project is defined to migrate the environment to Linux servers.

Rolling Cluster Upgrades

The online documentation for Novell Cluster Services for Linux describes an upgrade path called a rolling upgrade. This approach is to remove a NetWare server from the existing cluster and remove it from the eDirectory environment and install Linux on the same hardware. The Open Enterprise Linux server is then added to the NetWare cluster, and this process is repeated until all cluster nodes are upgraded.

The technical approach we use in this section is identical to that procedure, except that we do not start with deleting NetWare nodes, we start with adding new Linux servers. Especially for smaller environments, this might be a better approach, depending on the hardware requirements for the new cluster environment. With only two or three nodes, the level of fault tolerance could fall below an acceptable limit.

If you do not plan to buy new server hardware for your cluster and the hardware being used to run NetWare can be used for running Linux, you could also perform a regular rolling upgrade. In that case you start by deleting a NetWare node first and then start with our first step: installing the first Open Enterprise Linux server into your cluster.

In both scenarios the cluster effectively remains available to your end users and service is not interrupted.

Preparations and System Requirements

Before adding new cluster nodes to an existing NetWare cluster, you must meet the system requirements and perform some preparatory steps. All existing cluster nodes must be the same version of NetWare 6 or NetWare 6.5. The new Linux servers that will be added to the cluster must be fully configured and added to the tree. They also must have the software packages installed for Novell Storage Services and Novell Cluster Services. The installation of these components is discussed in detail in Chapter 4, "Installation and Configuration."

Another requirement is that the new Linux nodes have access to the shared storage, either through the Storage Area Network (SAN) or via iSCSI. If the latter is the case, also make sure that all iSCSI connections are up and running before starting the cluster services configuration. How to configure iSCSI is described in Chapter 7, "iSCSI."

Last but not least, it is important to have eDirectory replicas in place that are needed for the cluster node to work efficiently or, depending on the application, to work at all. More information about eDirectory design guidelines can be found in Chapter 3, "Clustering Design." The simple rule is that cluster nodes must have local access to objects that they need in order to function, such as the user objects of those that access the cluster.

Adding the First Linux Node to the Netware Cluster

As we described before, it is also possible to first remove a NetWare node from the cluster and reuse that hardware for this first server. Whatever method you choose, the steps are the same, and we have decided to first add a Linux node to the cluster. This procedure is also described in Chapter 4, but we will give you the short walkthrough here. We expect that at this point the new cluster node is part of the tree, has NSS and NCS installed, and has access to the shared storage.

Perform the following steps to add the server to the cluster:

1. | On the Linux server start YaST.
2. | In the System category select Novell Cluster Services.
3. | When prompted to also configure NSS, if you have not already done that previously on this server, select Yes to configure that component too. It is a mandatory component to be installed on an NCS cluster server.
4. | When prompted, enter the credentials for the administrative user.
5. | Enter the full distinguished name of the cluster where this node will be added.
6. | The IP address of this node will be displayed. Typically, this will be the primary IP address, unless more than one interface is configured. Select the address from the subnet where the other cluster nodes are. Note You can reuse the IP address of a removed NetWare node, but in that case upgrade one node at a time.
7. | Leave the check box to Start Clustering Service Now enabled and click Next.

The new node will now join the cluster. When the configuration is finished, the configuration utility will be closed automatically and you will be returned to YaST. Open a terminal window on the Linux server and execute the cluster view command (as root or other superuser) to check that the new node is online in the cluster. You can also execute the sbdutil -v command to check that the new node has been added to the cluster. This command is equivalent to the SBD VIEW ALL command on NetWare.

At this point you can either add new Linux nodes to the cluster or first remove one of your NetWare nodes. It depends on your upgrade scenario which order you choose, as long as you keep in mind that you need to provide fault tolerance and high availability to your end users.

Removing an Oes Netware Node From the Cluster

There is no fully automatic procedure for removing a node from a cluster. It is as simple as deleting the cluster object after disabling the cluster software.

Although leaving the cluster would automatically migrate any resources to other nodes, we advise you to migrate the resources manually. This gives you the chance to run them on the node of your choice and to watch the process more closely, resource by resource, which is harder when everything happens automatically at the server. Perform the following steps to remove a node from the cluster:

1. | On the NetWare server console, type cluster leave.
2. | Execute ULDNCS to unload Novell Cluster Services.
3. | In iManager open the eDirectory Administration task and select to delete an object. Browse to the cluster container and delete the cluster node object of this node. Tip If you have problems removing the object from iManager, log in to iMonitor for your server and click the iMonitor logo at the upper-left corner. In the configuration screen that appears you can now enable the advanced mode. Browse to the object in iMonitor, and in the left panel you'll see an extra Advanced Menu option, which allows you to delete the object.
4. | Remove the command that loads Cluster Services (LDNCS) from AUTOEXEC.NCF.

At this point you can decide what to do with this server. If you are reusing the hardware to install another Linux node on the same box, you can leave the server attached to the shared storage and start the Linux installation. Otherwise, if the server will be reused as a NetWare server in the same eDirectory environment, disconnect the server from the shared storage, and reboot the server.

You now can add other Linux nodes to the cluster or, when you have removed the last NetWare cluster node, go to the next section to finish the cluster upgrade.

Finalizing the Netware-to-Linux Cluster Upgrade

When all NetWare nodes are removed from the cluster and only Linux nodes are left, it is time to clean up and convert the cluster to a real Linux-only cluster.

During the period that the cluster is in mixed mode, the resource load and unload scripts are stored in the directory /etc/opt/novell/ncs. When changes are mode to the resource scripts, these files are automatically updated. When all NetWare cluster nodes have been removed, the Linux load and unload scripts can be moved into the corresponding eDirectory objects.

To finalize the cluster upgrade, follow these steps:

1. | Run cluster convert preview resource_name at the server console of one Linux cluster node. Replace resource_name with the name of a resource that you want preview. This will show you what modifications will be made to the resource scripts. The preview command can work only with individual resources.
2. | Run cluster convert commit at the server console of one Linux cluster node to finalize the conversion. This will convert all resources in the cluster.

The cluster container now contains only one legacy set of objects: the NetWare cluster resource template objects. When you're installing a new Linux cluster, these templates would have been generated for you, but when you're adding Linux nodes to a NetWare cluster, the old templates remain intact.

You can create the Linux-specific cluster resource templates by executing the following command on one of your Linux nodes:

/opt/novell/ncs/bin/ncs-configd.py -install_templates

In addition to generating Linux cluster resource templates, this command deletes all NetWare cluster resource templates. Because of this, use this command only after all nodes in the former NetWare cluster are converted to Linux.

Now that the entire cluster is upgraded, all applications can be upgraded to run in the new Linux environment.

Migrating to a New Cluster

All upgrade methods explained to this point were in-place upgrades and the mixed-mode upgrade in which data also remained in place in eDirectory (cluster configuration) and on the file system (files and rights). Sometimes it is not possible or wanted to perform an in-place upgrade. In those cases there is always the possibility to use a migration method to move your environment to new servers and to a new cluster.

The methods available for migrating servers to a cluster are not much different from those for migrating a single server. There are a few extra steps in a clustered environment that we will discuss with each available method.

Novell Server Consolidation and Migration Toolkit

Starting with Open Enterprise Server, Novell has bundled two migration utilities into one toolkit. The first utility already existed as the Migration Wizard, and the other one is called the Server Consolidation Utility. The first one is not really usable in a clustered environment. It can be used to upgrade the hardware of a server with which the server on the new hardware keeps the same identity throughout the entire eDirectory environment.

Note

The Migration Wizard replaces one server with another server, whereas the Server Consolidation Utility replaces one or more source servers with one or more destination servers.

Tip

Both utilities are available for all versions of NetWare and can also be used for servers that are of the same version, such as NetWare 6.5 to NetWare 6.5. The toolkit also supports migrations from Windows to NetWare and OES Linux.

This is important for security components, user home directories, and so forth that point to an NCP server object. And there are server applications and maybe scripts or other control files that point to that specific server. All of this is not an issue with a cluster because in general these configurations will point to a cluster virtual server. So to upgrade a server to new hardware, it is simple to remove the old server from the cluster and install the new server into the cluster as a cluster node. You probably want to do it in that order because of cluster licenses that allow only a maximum number of nodes in your cluster.

There can, however, also be scenarios in which you want to replace an entire cluster. Or when you want to consolidate existing servers into a cluster. Performing such a migration is not as simple as just copying the data. In a networking environment there are at least the following configuration items that also need to be copied or modified:

  • File-system rights

  • Disk and directory quotas

  • User home directories

  • Login scripts, batch files, and such

  • Application references, including UNC paths

The utilities that Novell delivers for migrations do not offer support for all of these configurations to be copied or changed. The Migration Wizard is the most complete because it places the new server in the identity of the old one, which keeps all server references and items named in the preceding list intact. But there is also the Server Consolidation Utility, which does not take care of everything. This utility merely copies the data from one server to another server, including the file-system rights. If you are moving data from one tree to another tree, it is also possible to copy the user objects to the new tree. There could be some objects that still would not be copied, such as ZENworks application objects and their user references.

Using the Server Consolidation Utility

The Server Consolidation Utility is part of the Novell Server Consolidation and Migration Toolkit. The software is located on the NetWare operating system CD-ROM (SCMT.EXE in the /products/Migration_Utilities directory). The utility runs from a Windows workstation.

It is not a topic of this book to explain how this utility works in detail. And for the most part the software is self-explanatory. As you can see in Figure 10.1, there are two panels that show the source and destination. Dragging data from the source to the destination volumes will structure the new environment. When the preparations are finished, a verification can be run, and when everything is guaranteed to work as planned, the actual data copy can be started.

Figure 10.1. Data can be copied from one cluster to another cluster with the Server Consolidation Utility.

[View full size image]

The process of copying data from one server to another server is the same for standalone servers as for cluster servers. In a clustered environment you drag and drop the files to or from the cluster virtual server and its volumes.

Tip

When migrating to an OES Linux cluster, the source server might not be able to connect to the Storage Management Data Requester (SMDR) on the target server if that server is a cluster node. The project verification process will generate an error message if this problem occurs. In that case restart the SMDR daemon so that it loads the configuration of the virtual server and its resources. Execute this command to restart the daemon: /etc/init.d/novell-smdrd restart.

There are still a few items in your environment that will not be copied to the new servers, whether standalone or in a clusterfor example, user home directories that are not updated. The next section describes how simple tools can be used to update all necessary objects.

The Real Wizardry: Post Migration Tasks

Copying the data and file-system rights from one server to another is one thing; updating any references in your environment that may still point to your old server to the new server is a completely different thing.

First of all, it is important to understand how these server references work. Your NetWare or OES Linux server is represented in eDirectory by an NCP server object. Other objects will contain attributes that point to that server object. Those attributes, also called properties, are not simple text fields that contain the server name; they are attributes of the type Distinguished Name that reference the actual object, regardless of its name. Because of this, when you rename a server all these references will automatically point to that renamed object. When you have an old server in your tree that is called FS1 and a new server called FS2, deleting FS1 and renaming FS2 will not update all references to server FS1. At the moment object FS1 is deleted, all references are lost. It is therefore very important to always update all references from the old server to the new server and only if all steps are completed to delete the old server from the tree.

A simple example of how server references can be updated to a new server is that of the Default Server property of users. Changing that to a new value is simple.

1. | Go into ConsoleOne or iManager and perform an advanced search for all users with the given Default Server property.
2. | In the results window, select all objects and choose Properties on Multiple Objects from the File menu.
3. | Next, select the new Default Server property for these users and click OK to apply this setting to all the users who had the old server as their default server.

All objects that pointed to the old server will now point to the new server.

For user home directories it is not as simple as this. One reason is that it is a unique value for each user. They all have a common part, the server volume, but also a personal part, the actual directory. When data and users are being copied to another tree with the Server Consolidation Utility, a template object can be used to set the home directory attribute in the new tree. But when the server and data stay in the same tree, nothing will be updated.

User home directories can be modified to point to a new location in several ways. There are third-party tools available for purchase that can help you with the most complex scenarios for user home directories. But also with some simple tools you already have, much is possible.

Tip

There is a great product available from Novell called File System Factory. It allows for policybased management of home directories. When you have to move users on a regular basis, such as in case of a school at the start of each semester, this tool will make your job easier. More information can be found at www.novell.com/products/filesystemfactory.

One of the methods you can use to change the Home Directory attribute is to export it via LDAP and, after making the necessary modifications in the export file, import the information again. Creating an LDAP export can be done via the Import Conversion and Export (ICE) Wizard in ConsoleOne and iManager. There is also an ICE command-line executable available for NetWare, Windows, and Linux. Because of that platform-independent availability and the enormous flexibility of the command-line tools, we will show you the command that exports the information from an LDAP server. ICE can be found in the ConsoleOne directory under the bin directory.

The following command should be entered on one line (the arrows are simply code continuation characters):

[View full width]
ice -S LDAP -s 10.0.0.1 -p 636 -d cn=admin,o=oes -w novell -F (objectclass=user) -a
   ndsHomeDirectory -b o=OES -L f:\public\rootcert.der -D LDIF -f c:\temp\userhomedir.ldif

This command exports the information from the server at IP address 10.0.0.1 via secure port 636 by using the certificate in f:\public\rootcert.der. The export function is performed as user Admin in the OES container, using the password novell. Only user objects are searched, -F (objectclass=user), and the search is performed, starting at the base of DA (-b o=OES). The only attribute that is exported is ndsHomeDirectory. The -S LDAP parameter tells the command that the operation uses LDAP as a source, and -D LDIF tells the command to use LDIF as the destination. In this case, it is writing the information to c:\temp\userhomedir.ldif.

The result of the LDIF operation should look like the following:

version: 1
dn: cn=Rob,o=OES
changetype: add
ndsHomeDirectory: cn=SRV1_DATA,ou=SRV1,o=OES#0#Users\Rob

You can use your own favorite tool to modify the LDIF file. For a not-too-complex modification, you can even use a simple text editor and use the replace function to change the server name. However, if your modification is more complex, such as adding a directory path to every user's setting, an editor that supports using macros is probably a better choice. Or use the powerful Linux commands in combination with regular expressions to modify the file.

The modified file, which imports the new home directory location into eDirectory, should look like the following:

version: 1
dn: cn=Rob,o=OES
changetype: modify
replace: ndsHomeDirectory
ndsHomeDirectory: cn=SRV2_DATA,ou=SRV2,o=OES#0#Users\Rob

In this scenario the users are being moved from server SRV1 to server SRV2. The references to the volume on the source server have been modified to point to the destination server.

The import command for this new LDIF file is the following:

[View full width]
ice -S LDIF -f c:\temp\userhomedir.ldif -D LDAP -s 192.168.1.101 -p 636 -d cn=admin,o=OES
   -w novell -L D:\public\rootcert.der

The command is constructed in the same way as the export command, except that we now use LDIF as the source and LDAP as the destination. In addition, we do not have to specify any search parameters because the full distinguished names are in the LDIF file.

Note

Before modifying the home directories, make sure that the files and trustees are also in place. Otherwise, your users will experience problems during the login process.

User home directories are a very common configuration item in many NetWare environments. That is the reason we described how to modify them in depth. There could be many more items that need to be modified, ranging from login scripts and batch files to ZENworks application objects that may contain UNC paths. If there are any other objects in eDirectory that contain a reference to the old server, there is a chance you can find them with the following export command:

[View full width]
ice -S LDAP -s 10.0.0.1 -p 636 -d cn=admin,o=oes -w novell -F (objectclass=*) -a * -b
   o=OES -L f:\public\rootcert.der -D LDIF -f c:\temp\allobjects.ldif

If you search the exported file for the old server name, you might find all entries that still point to that server. There is no guarantee that you will find all references because some attributes are not exported though LDAP, such as login scripts.


Chapter 11. Using SUSE Linux Enterprise Server Clustering Options

This book is about Novell Cluster Services (NCS), and that's not the same as the native Linux clustering options included in SUSE Linux. Because, however, in some circumstances it may be useful to use these native Linux options anyway, this chapter provides a quick guide to the setup of such a cluster


Heartbeat Versus Novell Cluster Services

Without any doubt, in large business environments NCS is the better choice to provide high availability for business-critical services. There are various advantages offered by NCS as compared to the Linux Heartbeat software. Among these advantages, one of the most important is the inclusion of the entire configuration in eDirectory; therefore, the administrator can manage the entire configuration of his cluster from one central point in the network. Another major advantage is that NCS offers support for up to 32 nodes, whereas at the moment this was written, support for more than 2 nodes was not stable yet in the Heartbeat software. Other advantages of NCS over Heartbeat are support for the NSS file system, support for mixed clusters to migrate from NetWare to Linux, and, of course, Novell's strong support for this software by its worldwide support organization.

Despite of all these advantages, in some scenarios an administrator might want to choose native Linux clustering services. One reason to do this may be that high availability is needed for a service at a branch office where eDirectory is not used. In a pure Linux environment, Heartbeat may also be the better choice because it is free software that doesn't require any licenses to install.

The Heartbeat package is a good choice to deliver such a high-availability solution. You should, however, be aware that Heartbeat is not the only option available for Linux clustering. It is, however, a very stable cluster solution that is included with SUSE Linux Enterprise Server by default, and for that reason it is covered in this chapter. The goal of this chapter is not to give a complete overview of all options available with the Heartbeat software, but to give the administrator who wants to implement it a quick start to set up such a cluster.


Installing SUSE Linux High-Availability Components

In SUSE Linux Enterprise Server, all software to create a two-node cluster is available by default. We will first cover how to install this software:

1.
From the SUSE Linux Enterprise Server console, start YaST and enter the password of the root user.

2.
Select Software, Install and Remove Software.

3.
From the Filter option, choose Selections. Next, make sure that the option High-Availability is checked as shown in Figure 11.1. This will select all software modules to install and manage a cluster based on the Heartbeat packages.

Figure 11.1. The High-Availability selection includes all software needed to run a Heartbeat cluster.

[View full size image]

4.
Click Accept to install the required software.


Configuring the Shared Storage with a DRBD

For sure, you can run a Heartbeat cluster using a storage area network (SAN) or iSCSI as shared storage. For a simple setup, however, this may not be the most appropriate way to set up the shared storage, because expensive hardware is needed to configure the storage device. An easy yet stable method of providing a shared storage solution for the Heartbeat cluster is by configuring a Distributed Replicated Block Device (DRBD). The advantage is that no external storage device needs to be purchased, but a local partition or disk on the nodes in the cluster can be used for shared storage. Alternatively, if the service you want to cluster doesn't need a shared storage solution, it is perfectly possible to configure the Heartbeat cluster without shared storage. In most cases, however, it is more practical to provide some shared storage solution, if only to make management of configuration files easier. This chapter discusses how to set up such a shared storage device. The only requirement for the setup of this shared storage is that both nodes be cluster nodes. If the service fails on the primary node, it will be activated on the other node. The Heartbeat load script will automatically mount the shared device in read-write mode if that happens.

Basically, the DRBD is a RAID 1 solution running over the network. In this configuration one node has read-write access to the DRBD device and the other node does not. For sure, it makes sense to give read-write access to the node that currently has the clustered service running. To use the DRBD device, a specific device has to be created. This involves using mknod to create the device, but also to create some configuration files. Finally, the DRBD service that gives access to the device has to be started, and one of the nodes needs to be assigned as the primary device. Here's how to set up a DRBD device:

  1. After installation, it is a good idea to copy the sample DRBD configuration file in /usr/share/doc/packages/drbd/drbd.conf to /etc/drbd.conf and edit this file. There is a lot of comment in this file that explains perfectly how to configure the DRBD device. Make sure that the configuration file has the same contents on both servers; otherwise, the DRBD device would be available from just one server.

  2. For a simple setup to provide the DRBD device, the following parameters can be used. Note that this is just a workable example; your environment may require a different setup. Consult the drbd.conf man page for more information on how to set up the DRBD device. The following example includes some workable code you can use to set up a DRBD device:

    resource rdbd0 {
   protocol C;
   startup {
      wfc-timeout 0;
      degr-wfc-timeout 120;
   disk {
      on-io-error detach;
   }
   syncer {
      rate 100M;

      group 1;
      al-extents 257;
   }

   on server1 {
      device /dev/drbd0;
      disk /dev/sdb1;
      address   192.168.0.201:7788;
      meta-disk   internal;
   }

   on server2 {
      device   /dev/drbd0;
      disk   /dev/sdb1;
      address   192.168.0.202:7788;
      meta-disk   internal;
   }
}

The foregoing script consists of two parts. In the first part, some generic DRBD settings are configured. These settings are applied to the device drbd0, which is specified from the resource drbd0 setting. Next the protocol that has to be used is defined. There are three choices here: protocols A, B, and C. Protocol A is the fastest, protocol C the most secure; so make your own choice as to which is more important to you: speed or security.

Note

When protocol A is used, write I/O is reported as completed, if it has reached the local disk and the local tcp-send buffer. Therefore, if after the I/O has reached the send buffer the network connection drops, data will get lost. With the protocol B setting, write I/O is reported as completed if it has reached the local disk and the remote buffer cache. This ensures that the data at least has reached the remote computer. If protocol C is used, write I/O is reported as completed if it has reached both the local and the remote disks; therefore, protocol C is considered the most secure protocol setting.

Next, some timeout values should be configured. The first parameter, wfc-timeout 0 specifies the amount of time that the init script blocks the boot process until the resources are connected. By default this value is set to 0, in which case the init script doesn't care about the actual state of the DRBD device. Next, the degr-wfc-timeout parameter specifies how long to wait for connection timeout if this node was a degraded cluster. The default value specifies that it has to wait for two minutes. This allows the other node in the cluster to come back and hand over control to the device.

Next the disk parameter on-io-error detach specifies what has to happen if the DRDB device reports an error. The option detach specifies that the node drops the storage device and continues in diskless mode.

The syncer parameters are the last set of generic parameters, and not the least important. If you forget them, the performance of synchronization between nodes is worse than lousy, so use these parameters to specify how synchronization should take place. The synchronization parameters can be used to specify the amount of bandwidth that can be used by the DRBD process. The parameter rate 100M in the example sets the maximum amount of bandwidth for synchronization of the device to 100Mb. If you do have a 100Mb network and you want other traffic to happen as well, this is not a good choice; you should set it lower so that other processes have some bandwidth as well. The parameters group 1 and al-extents are used to define the number of physical extents that can be synchronized at the same time. Each physical extent is 4MB, so a total of 257 will allow synchronization of about 1GB at the same time. Basically, the value for the parameter al-extents should reflect the number of 4MB blocks needed to cover synchronization of your DRBD device; for 4GB a value of 1024 would be a good value, for example.

Next, there are the sections that define how both DRBD nodes should be configured. The first parameter specifies the name of the DRBD device. Then the next parameter specifies the name of the internal device on the server that should be used as a DRBD device. It makes life easier if the device names are similar on both nodes, but this is not a requirement. Next, the address and port numbers need to be specified. Last, the meta-disk parameter is used to specify where metadata should be written. Normally, the device specified with the disk parameter is used for this purpose. Be careful that nothing else is stored at the device mentioned because it will all be gone after the DRBD device starts writing metadata to this device.

After creating the drbd.conf file, you have to create the DRBD devices. For DRBD to work, you need a corresponding block device in the directory /dev; otherwise, it won't work. Actually, there are two different devices that can be used for DRBD: One is the drbd device with major device ID 147, and the other is the NB device with device major number 43. In the current release, the DRDB devices are recommended for usage, so just forget about the NB devices. You can use the mknod command to create the DRBD devices. The command that can be used for that is shown here (be sure to execute this command as root, or else it will not work):

for i in $(seq 0 15); do mknod /dev/drbd$i b 147 $i; done

Make sure that this command is executed on both nodes where the DRBD device must be used; it will ensure that 16 DRBD device files are created. When this has happened, use the rcdrbd start command on both servers to start the DRBD services. If this happens without any obvious error messages, use the command cat /proc/drbd to monitor that the DRBD device is successfully created on both devices (see Figure 11.2). Sometimes there may be an error message when loading the DRBD device, stating that the device is currently busy. If that happens, we recommend that you try again in a few momentsmost of the time that helps. For serious troubleshooting, use tail /var/log/messages for error messages that may indicate why the DRBD device cannot be started.

Figure 11.2. Use cat /proc/drbd to monitor whether the DRBD device has loaded successfully.

[View full size image]

Now that you have verified that the service has started successfully, you need to make sure that it starts automatically the next time you boot your servers. To insert the service to all relevant runlevels on your servers so that it will be booted automatically, use the insserv drbd command on both servers. The current state of the DRBD service is that it will be started on both servers automatically. At this stage, however, it is not usable. To make it usable, you need to configure the Heartbeat software in such a way that one of the servers is configured as the primary server, while the other is used as the secondary. In the next section you'll read how this needs to be done.

Note

We have covered DRBD as an easy storage solution in this chapter. As an alternative, the software from the iSCSI target project for shared storage can be used. Currently, the only thing preventing the use of this software is the requirement of the 2.6.13 kernel. This problem, however, will be fixed in the future. Consult Chapter 7, "iSCSI," for more information about iSCSI configuration.


Clustering the NFS Server with Heartbeat

Basically, the configuration of a Heartbeat cluster is a five-step procedure:

1. | Edit the sample configuration files.
2. | Use the authentication keys to ensure secure communications.
3. | Tune the main configuration file /etc/ha.d/ha.cf on both nodes.
4. | Use the configuration file haresources to configure shared resources in the Heartbeat network.
5. | Start the Heartbeat software on both nodes.

We will discuss the configuration of the Heartbeat software based on an example. In this example, the NFS server will be configured for high availability. The server will use the DRBD device as the shared storage device.

Edit the Sample Configuration Files

To make it easier for you to configure the Heartbeat software, some sample configuration files are provided. An easy way to get started is by copying these configuration files to the directory /etc/ha.d on both servers. You can find these files in the directory /usr/share/doc/packages/heartbeat. Three of these files are of particular interest:

  • authkeys: This file is used to secure communications between nodes with authentication keys.

  • ha.cf: This is the main configuration file in which default behavior of the Heartbeat software is specified.

  • haresources: In this file, the shared resources are defined.

Use Authentication Keys to Ensure Secure Communications

On a secure network, it is not of highest importance, but if traffic between nodes has to cross less secure links, an important option is the usage of authentication keys to secure the communication between both nodes. The structure of this file is really simple; just two lines of configuration are needed. In the first line, you specify which type of authentication needs to be used. For this purpose, you can choose from crc, sha1, and md5. Crc is the ideal solution for a network that has already been secured by other means; it doesn't offer any real security and therefore is the fastest option for use on a secure network.

The other two options are sha1 and md5. Both are secure, so it doesn't really matter which one is used. In the code listing that follows, you see all relevant lines that have to be created in /etc/ha.d/authkeys to ensure that authentication can take place in a secure manner:

#
auth 3
#1 crc
#2 sha1 secret
3 md5 novell

In this example, you can see that the type of authentication that has to be used is referred to on the line auth 3. The number 3 refers to the third line in the configuration file, where md5 authentication is configured for the use of a shared secret. After creating this file on one of the nodes, you can copy it to the other node. Use scp as in the following example to do this:

scp /etc/ha.d/authkeys othernode:/etc/ha.d

As a last step, you need to ensure that the permission mode for the files is set to 600 on both nodes. Use the following command to accomplish this task:

chmod 600 /etc/ha.d/authkeys

Tune the Main Configuration File ha.cf on Both Nodes

One of the major tasks in successfully configuring the Heartbeat cluster is to configure the main configuration file ha.cf. In this file the conduct of the Heartbeat software is defined. The file should be more or less the same on both nodes, although some parameters are node specific. The first parameters you have to enable look rather trivial, but will make you happy later; these are the parameters to enable logging. Use all logging mechanisms; it will make it easier for you later to troubleshoot why the software isn't acting the way you would like it to. Be sure to remove the hash sign used to comment out some of the following three lines to enable full logging:

debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility local0

Next, you need to specify how often heartbeat packages are sent across the network and when the other node will be considered dead. The following lines are used to specify this:

keepalive 2
deadtime 30
warntime 10
initdead 120

Of these parameters, the keepalive parameter specifies that a heartbeat message should be sent every 2 seconds. In some environments, this will absolutely be too long, so you could consider setting it lower, for example, to 500ms. The next parameter is very important as well: The deadtime parameter defines when a service will officially be considered dead and services will start to fail over to the other node. By default, this is only after 30 seconds. For some environments this absolutely is too long, so you should consider setting it lower. Then there is the warntime parameter, which specifies when the first warning about a service being not available is sent to the log files. Last, the initdead parameter specifies the timeout value used after the services are first started. The default value will give the other node 120 seconds to come up and activate its resources. If it hasn't come up after this period, the other node will be considered dead and services will start on the remaining node. Because it can take some time for another node to come up, it is recommended to give this parameter a rather high value.

The last options that need to be set in ha.cf are the names of the cluster nodes. The names of the nodes included here should be exactly the same as the names of the nodes displayed when using the uname -a command on these nodes. These parameters could, for example, look like the following:

node server1
node server2

Apart from these required options, many other options can be used as well. For example, there are parameters to define the failover and failback behavior of resources in the network, and there are parameters to define how the heartbeat packets should be sent in the network. Consult the ha.cf man page for more information about these options.

After creating the ha.cf file on either one of the nodes in the cluster, be sure to copy it to the other node. You can do this using the scp command as shown here:

scp /etc/ha.d/ha.cf othernode:/etc/ha.d

Create Shared Resources by Editing the haresources File

Now that the generic parameters for the Heartbeat software have been configured, it is time to define the shared resources on the network. This is done in the configuration file /etc/ha.d/haresources. In this file the following tasks have to be accomplished:

  • Define the name of the primary node where the resource should run

  • Define the IP address on which the shared resource can be used

  • If used, specify the name of the shared device that should be used

  • Specify which service or services should be loaded as shared resources

Following is an example of a line used to load the nfsserver as a shared resource:

drbd1 192.168.0.23 datadisk::drbd0 Filesystem::/dev/drbd0::/data::reiserfs nfsserver

Defining the Name of the Primary Node

Each line where a cluster resource is defined starts with the name of the primary node. The resource will always try to load on this resource. If this primary node goes down and then comes up again, the default behavior is that the resource automatically fails back to this primary node.

Defining the Ip Address of the Shared Resource

The only mandatory element of all shared resources is the IP address of these resources. Each resource should have its own IP address on which it can be reached. This should be a unique IP address. The server hosting the resource will activate this IP address as one of its virtual IP addresses, as shown in Figure 11.3.

Figure 11.3. Each shared resource has its own unique IP address displayed on the server hosting the resource as a virtual IP address.

[View full size image]

Specifying the Name of the Shared Device

If a resource uses a shared device, all information should be present in /etc/ha.d/haresources to specify how this device should be loaded. For this purpose, two scripts can be used, which are both in the subdirectory /etc/ha.d/resource.d. First of these scripts is the datadisk script (which actually is a link to the drbddisk script). This script has an important task: It activates the DRBD device on the host where the resources will be loaded. While loading the DRBD device on this host, the datadisk script will make it the primary device on this host. In this way, the other host cannot write to the DRBD device and all data written to the device will be synchronized immediately to the other node. In the line where the shared resource is defined, you can see that the datadisk script takes one argument, which is the device name of the DRBD device. To separate the name of the script from the names of its arguments, two colons are used.

After specifying what device is used as the shared storage, next the resource load script needs to define how that device is to be activated. Note that this task has to be connected to the loading of the resource on a server and cannot occur in /etc/fstab; if it would be handled in /etc/fstab, it would be impossible to migrate the shared device from one node to the other. To mount the shared device, the Filesystem script that is in /etc/ha.d/resource.d is used.

This script takes three parameters: the complete device name of the shared device, the directory on which the shared device must be loaded, and finally the name of the file system that must be used to mount the device. Here also, the script is separated from its arguments by using two colons as can be seen in the example where the resource for the NFS server is created.

Specifying Which Services Should be Loaded

Last, you need to specify what service should be loaded as the clustered resource. Here you can specify the name of just one service, but it is possible as well to load more than one service from one line. By default, the Heartbeat software will try to locate the name of this script from /etc/init.d and /etc/ha.d/resource.d, so make sure that the script you refer to is present at one of these locations. This script will automatically be started with the start argument, and it will be stopped with the stop argument. Therefore, the script you refer to should listen to these arguments.

Note

You have just learned how to configure a complex resource for use in the Heartbeat cluster environment. Configuring a shared resource can be much easier. It is, for example, possible to configure just a secondary IP address as the cluster resource, or just the shared storage device without anything else, or just the service you want to cluster without using any shared storage. For example, the following line is perfectly legal (although not extremely useful):

     server1 192.168.0.204

Also, the following can be used to configure a clustered resource that doesn't need any shared storage because all configuration it requires is available on the local file system:

     server1 192.168.0.204 dhcp

Start the Heartbeat Software on Both Nodes

When all configuration has been done, it is time to start the Heartbeat software on both nodes. Use the rcheartbeat command to do this. We recommend while doing this to monitor /var/log/messages with the tail -f command. It will show in an excellent way whether loading of the Heartbeat software and the shared resource is successful. An example of this is shown in Figure 11.4.

Figure 11.4. /var/log/messages will show precisely whether loading of the shared resource was successful.

[View full size image]

Configure NFS for Use in the Heartbeat Cluster

The Heartbeat cluster is now configured the right way. To get the NFS server running properly, however, some minimal action is required. Most important is that the directory /var/lib/nfs, which is used by default for file locking by NFS, is replaced with a location on the shared disk. If you don't do that, file locks can't migrate when the NFS service is migrated. The best way to make information about file locks available at the other server is by making a symbolic link on the shared storage, pointing to the real location of the files in /var/lib/nfs. The following procedure shows how to do this if the shared disk is mounted on the /data directory:

1.
Unload the Heartbeat services on both servers by using the rcheartbeat stop command.

2.
Make sure that on the server that is supposed to act as primary in the cluster, the shared storage device is accessible and mounted on its mountpoint. You don't need to mount it on the secondary server (even if you wanted to, you couldn't, because only one server at a time can access the shared device).

3.
On both servers, make a symbolic link from the location of the NFS lockfiles to the directory on the shared storage device where you want to make them accessible. If, for example, you want to store the lockfiles in /data/nfslock, use the following command on both servers to create the link:

ln -s /var/lib/nfs /data/nfslock

Make sure that when you do this the mount is not up. Otherwise, you risk that one of the servers will complain when you try to create a link because the link already exists on the mounted device.

4.
Dismount the shared device.

5.
Modify the haresources file to load the nfslock daemon as well. A line like the one in the following example will make sure that it is loaded properly:

server1 10.0.0.2 datadisk::drbd0 Filesystem::/dev/drbd0::/data::reiserfs nfsserver nfslock

6.
Start Heartbeat services on both servers.


Managing the Shared Resource

Now that you have the Heartbeat cluster running, it is good to know that some commands are available for management and monitoring of the cluster and its resources. Don't expect extensive management capabilities like you have them with NCS; some basic commands, however, are available. Most of them are used as a parameter from the cl_status command. Following is a summary of the most important options:

  • cl_status hbstatus: Shows whether the Heartbeat services are running on the local computer

  • cl_status listnodes: Shows all nodes currently in the cluster

  • cl_status rscstatus: Shows the state of clustered resources

Note

The Heartbeat package currently does not include software that actively monitors the current state of the clustered resource. For this purpose, you can write a shell script that monitors whether the service is still running and alerts if this is not the case. Alternatively, the mon utility can be used. This utility, which is not installed by default, uses SNMP to monitor the availability of services. Because it is beyond the scope of this book, the mon utility is not discussed in this chapter. For more information, we recommend Karl Kopper's excellent book The Linux Enterprise Cluster.


Avoiding Split Brain

One issue has not been dealt with so far. In a cluster environment, it can happen that more than one node in the cluster thinks that it has exclusive access to a cluster resource. Such a configuration is known as split brain. If this resource is the shared storage device, the results can be severe; if, for example, a database is stored on the shared disk device, the entire database may become corrupted. Therefore, in a serious clustered environment, precautionary measures must be taken to prevent such a situation. Some nice features are available with the Heartbeat clustering software to take care that the failing node is terminated when it fails.

One solution to cast off the failing node is to use a Stonith device. Stonith, which stands for Shoot the Other Node in the Head, refers to a technique that makes sure that the failing node is cast off efficiently. For Stonith configuration, a smart power device connected to both servers must be used. A serial cable or network cable running from both servers to this device allows the Heartbeat software to send commands to this device, which controls the power supply to the other server. If needed, the other node can reset a failing cluster node this way.

A Stonith device is used in the following way:

1. | The remaining server doesn't hear incoming heartbeat packages anymore and concludes that the other server fails.
2. | The remaining server sends the Stonith reset command to the Stonith device.
3. | The Stonith device turns off power on the failing server. This makes sure that a split brain situation is avoided.
4. | All resources that were running on the other server are failed over automatically.

Because each Stonith device has its own unique set of commands, it is not possible to give a "one size fits all" description on how to configure the device.

Tip

In a production network, it is recommended that you use a Stonith device. Your uninterruptible power supply (UPS) manufacturer can inform you about UPS models that support Stonith. If, however, for some reason you cannot lay your hands on such a UPS, two other methods are available as well. First is the meatware device (a human being) that is alerted to a failing node in the cluster. Second is the option to use SSH to send a command to the other node in the cluster. The latter method, however, is not recommended because there are too many situations in which it can fail.


Summary

In this chapter, you have read about the way in which the Linux Heartbeat software can be used to make Linux services highly available. You have learned how to set up a shared storage device using a DRBD, and you have read how to create cluster resources. Finally, a short description has been provided on how Stonith can be used to avoid split brain situations on the cluster.


Chapter 12. Introduction to Business Continuity Clustering

Having a high-availability cluster is a good thing to prevent disasters from causing major problems with the continuity of your business. Sometimes, however, one cluster is not enough. What if, for example, a flood or an earthquake wipes out an entire site? In that case an ordinary cluster won't help you. Especially for these kinds of situations, Novell has launched the Business Continuity Clustering (BCC) solution. This solution provides a true cluster of clusters. In this chapter BCC is introduced from a high-level overview perspective.


Business Continuity Clustering Overview

The goal of Business Continuity Clustering is to connect and synchronize information between clusters. At the time of writing this book, a maximum of four clusters is supported. In the 1.0 version, these need to be NetWare clusters; in BCC version 1.1, NCS 1.8 on Linux is supported as well. The clusters that are synchronized with BCC are often located at separate geographic sites. If one of the clusters goes down, the other clusters can take over the workload. In this way, nonstop access to mission-critical data and resources is guaranteed.

BCC is tightly integrated with eDirectory and synchronizes information between two to four clusters using eDirectory. For this purpose, Identity Manager, formerly known as DirXML, is used. Identity Manager is a Novell product that is used to synchronize information between different data sources. Originally, the purpose of Identity Manager was to synchronize, for example, between an eDirectory environment and an Active Directory environment, a PeopleSoft data store and many other applications, but it can be used to synchronize information within the same eDirectory structure as well. This is exactly the way BCC is using Identity Manager.

When a BCC solution is operational, services, configurations, and essential data can be migrated between two clusters with just one click. In fact, it is as easy as migrating a clustered resource within a Novell Cluster Services (NCS) environment where no synchronization with external resources takes place. In fact, BCC works a lot like NCS from an administrative perspective. There is, however, one major difference: Because it involves a lot of traffic and the results of a failover are rather severe, there is no such thing as automatic failover. You will always need to migrate BCC resources manually.

High Availability Between Clusters

Speaking in a generic way, there are two methods for setting a high-availability environment if clusters at different physical sites are involved. The first solution is the so-called stretched cluster. In this solution, one cluster is created in which the nodes in the cluster are located in different geographical areas. All nodes are in the same eDirectory tree and in the same cluster. In this solution, the cluster heartbeat is happening over the WAN connection. Also the storage area network (SAN) is available across the WAN, with a disk array at both sites and both disk arrays connected by a dedicated high-speed fiber connection. In the setup of the cluster, nodes at site B must be configured to take over services from site A if an entire site goes down.

There are, however, some limitations to this solution. The first is that in an NCS environment, all nodes in the cluster need to be in the same subnet. Therefore, in a stretched cluster, you would be forced to use the same subnet across the WAN. Also, the eDirectory in which the stretched cluster is created needs to stretch the WAN because all cluster objects need to be in the same eDirectory container, and this might complicate eDirectory design. Another disadvantage is that you are still working with just one cluster, and that doesn't offer the best possible fault tolerance; if you just have one cluster, you also have one single point of failure and that should be avoided.

A stretched cluster offers some advantages as well. Resources can fail over automatically, which is not possible in a cluster of clusters. Also, it is a lot easier to manage a stretched cluster than it is to manage separate clusters in a cluster of clusters.

The second solution to ensure high availability across sites is to build a cluster of clusters. In this solution, two different clusters are created and information is synchronized between the clusters, while both clusters are active at the same time. This is exactly what is happening in a BCC environment. In this scenario, there also needs to be replication of data blocks on the SAN; this is the responsibility of the SAN hardware. Other mirroring techniques also can be used to ensure availability of data across the different clusters in a cluster of clusters. For example, if no SAN is available, an rsync solution could be deployed to ensure data synchronization. If in a cluster of clusters, one of the sites would go down entirely, the entire cluster can fail over to the other site.

Note

Rsync is a solution that can be used to synchronize files or directories across the network. Rsync will be configured to synchronize files and/or directories at a specified time interval. It is not the best synchronization option available because delays will occur before changes are synchronized. For synchronization of small amounts of data, however, it is a good solution.

A cluster of clusters does offer some flexibility that is not available in a stretched cluster. These advantages are listed here:

  • The change of LUNs in the SAN is minimized.

  • There is no need to create eDirectory partitions that cross the WAN.

  • Each cluster can be in a separate eDirectory tree.

  • Each cluster can have its own subnet.

  • If one site fails, clustered resources can fail over to separate clusters. This feature is called multiple-site fan-out failover support.

  • There is no need to mirror SBD partitions between sites.

The most important disadvantage of the cluster-of-clusters solution is that resource configurations must be kept in sync somehow. In a BCC environment, Identity Manager is used for this goal.

Using BCC to Create a Cluster of Clusters

The main advantage Novell BCC has to offer as compared to a cluster of clusters is that synchronization of information between the clusters is happening automatically by using the Identity Manager eDirectory to eDirectory drivers. Novell BCC treats each of the separate sites in the cluster of clusters as a node in this metacluster. This allows an entire site to fail over to multiple sites. To add some level of automation to the failover process, BCC uses SAN hardware. The different Logical Units (LUNs) can be mirrored across the SAN. How exactly that happens depends entirely on the options offered by your SAN vendor. The entire configuration is managed with iManager, which allows for simplified management. If enhanced control and customization are needed, scripts can be used.

BCC Usage Scenarios

There are three scenarios in which BCC can be used:

  • A two-site Business Continuity Cluster

  • A multiple-site Business Continuity Cluster

  • A low-cost Business Continuity Cluster

A short description of these three scenarios follows.

The Two-Site Solution

If BCC is used in a two-site solution, two different scenarios can be deployed. First, the remote site can be configured as a passive site that takes over services only if the primary site fails. If services need to be used at both sites, both sites can be active as well. In this scenario, either site can take over services from the other site if needed; you should, however, be aware that enough resources need to be present at both sites to allow for a smooth failover. The scenario in which both sites are active is the more realistic scenario for large companies.

The Multiple-Site Solution

In a large environment, BCC can support up to 32 nodes per site with a maximum of four sites. In this scenario, if a site fails the clustered resources can fan out failover between sites. To replicate data across the sites, the solutions offered by the SAN vendors need to be used, although host-based mirroring solutions such as rsync can be used as well.

Low-Cost Testing Solution

Normally, a company would use a Fiber Channel SAN to ensure data availability across the different sites. If no Fiber Channel SAN is available, data block replication can be accomplished with iSCSI or rsync as well. This is ideal for test environments and environments where no budget is available for a Fiber Channel SAN; you should, however, be aware that such a solution does not offer the same performance level. In the rest of this chapter, you'll learn how you can create your own BCC test environment.


Building Your Own BCC Test Environment

This section describes how to build your own BCC test environment. Because when this was written, BCC 1.1, which supports Linux clusters, was not available yet, you will learn how to install BCC 1.0 in an Open Enterprise Server (OES) NetWare environment. With some modifications, this procedure will also apply to a BCC 1.1 environment. Before you begin, you must make sure that the following requirements are met:

  • NetWare 6.5 Support Pack 2 or later is installed on all nodes in the cluster.

  • Two to four clusters running Novell Cluster Services 1.7 or higher are installed.

  • Each cluster must have a unique name.

  • To keep it simple, we recommend setting up the test environment with both clusters in the same tree; this will simplify the Identity Manager setup a great deal.

The following procedure describes how to create a BCC environment based on two two-node clusters. In these clusters, iSCSI is used as the shared storage solution. As you will learn, in both clusters there is an iSCSI target that plays the role of the SAN in the environment.

Setup and Installation of NetWare and NCS

In this scenario, we will build one eDirectory tree with the name BCC-TREE, as shown in Figure 12.1. In this tree, two different Organizational Units (OUs) are created: site1 and site2. Both containers contain three servers, of which the first server functions as an iSCSI target. All other objects in the tree are not relevant for this setup.

Figure 12.1. eDirectory tree structure for the BCC test tree.

Follow this procedure to create a BCC test environment:

1.
Install Open Enterprise Server NetWare on all servers in the tree as illustrated in Figure 12.1. On all servers, make sure that at least Apache2 and iManager are installed. On bcc1-node1 and bcc2-node1, the iSCSI target software must be installed as well.

2.
After installation of the operating system, make sure that time is synchronized properly between all servers. Then put a R/W-replica on all servers. For the purpose of this lab setup, it is not necessary to create a partition for the different locations. In a real-life situation you might want to do that.

3.
If resources on your servers are limited, make sure that AFP, NFS, CIFS, and STARTX are disabled with a remark-sign in AUTOEXEC.NCF.

4.
Update the hosts file on all servers. In the tree as described previously, you can use the following entries:

192.168.2.10 | bcc1-node1
192.168.2.20 | bcc1-node2
192.168.2.30 | bcc1-node3
192.168.2.110 | bcc2-node1
192.168.2.120 | bcc2-node2
192.168.2.130 | bcc2-node2
192.168.2.50 | cluster1
192.168.2.150 | cluster2

5.
On the iSCSI target servers, load NSSMU and create three iSCSI partitions as in Table 12.1. Also make the appropriate server objects trustees of the iSCSI target object. Consult Chapter 7, "iSCSI," for more details on the iSCSI setup.

Table 12.1. Setup of iSCSI Targets

Server

Type

Size

Label

Trustee

bcc1-node1

iSCSI

20MB

SBD SITE1

All

iSCSI

400MB

DATA SITE1

All

iSCSI

510MB

MIRROR DATA SITE2

All

bcc2-node1

iSCSI

20MB

SBD SITE2

All

iSCSI

500MB

DATA SITE2

All

iSCSI

410MB

MIRROR DATA SITE1

All

6.
On all servers that are going to be used as cluster nodes and iSCSI initiators, create a file with the name INITSTRT.NCF and give it the following contents:

ION.NCF

ISCSINIT CONNECT 192.168.2.10
ISCSINIT CONNECT 192.168.2.110

ISCSINIT INFO

7.
Install the clusters for both sites. Complete the NCS installation procedure and check that NCS comes up on all servers. Next, remove the line where NCS is started by calling the LDNCS.NCF script from AUTOEXEC.NCF. This is because later in the procedure, you must make sure that the Identity Manager drivers have had enough time to synchronize before starting NCS or BCC.

Install Identity Manager 2.0

BCC uses Identity Manager 2.0 to synchronize information across eDirectory trees or even within an eDirectory tree. At this stage, you need to perform a basic installation of Identity Manager 2.0. Everything that is needed is provided with the free Identity Manager 2.0 starter pack, which is part of Open Enterprise Server or which can be downloaded free from http://www.novell.com/download. Complete the following installation procedure on one server in each cluster you want to add to the BCC environment:

1.
Mount the Identity Manager installation CD on bcc1-node2 (not on the iSCSI target, but on the first node that is a member of the cluster).

2.
From the graphical user interface on the server, click Novell and then select Install (see Figure 12.2). In the list of installed products, click Add and browse to the products.ni file in the NetWare section on the Identity Manager installation CD. Then click OK twice to continue. This will start the Identity Manager installation procedure.

Figure 12.2. The installation of Identity Manager is performed from the console of the NetWare server.

[View full size image]

3.
Click Next and then click I Accept to accept the terms of the license agreement. You will now see an overview of all components of an Identity Manager (formerly known as DirXML) environment. Notice that often the Identity Manager components are still referred to as DirXML components.

4.
Click Next to enter the components screen. Make sure that all components are selected and then click Next to continue.

5.
As shown in Figure 12.3, Identity Manager consists of an engine and drivers. The engine is always needed on an Identity Manager server; a driver is needed for each application that needs to be synchronized with. In this case, synchronization with external applications is not needed. Therefore, just select the eDirectory driver and click Next to continue. Both servers will need the DirXML Engine as well.

Figure 12.3. In a BCC environment, only the DirXML Engine and the eDirectory driver are needed.

[View full size image]

6.
Ignore the message about activation of Identity Manager components and click OK to continue.

7.
To extend the schema, enter the name and password of the admin user with sufficient permissions to perform this schema extension. This name must be entered in LDAP format, so enter cn=admin,o=bcc instead of .admin.bcc. Then click Next to proceed.

8.
From the Components screen, click Next. Two more components screens will appear. Leave the default selection and click Next to continue. Then click Finish to start copying the Identity Manager files to your server.

9.
If prompted that a newer file exists, choose never to overwrite newer files on your server.

10.
When the installation has completed, click Close.

11.
Restart your server when the installation of Identity Manager has completed.

Business Continuity Cluster Services Installation

Now that everything is in place and installed, just a few steps remain. First, BCC has to be installed. Then, the Identity Manager eDirectory drivers need to be configured to synchronize data between both clusters. After that, the final phase of the configuration can be completed: BCC has to be configured to enable failover between two to four clusters. In this section you'll learn how to install the BCC product.

Tip

A free evaluation version of BCC is available from www.novell.com/download.

To install BCC, apply the following procedure:

1.
Insert the BCC CD in a Windows workstation and run install.exe from the CD. Click Next and then I Accept to start the installation procedure.

2.
When asked what components to install, select Novell Business Continuity Cluster and iManager Snapins for Novell NetWare Based Management Servers, as shown in Figure 12.4. Then click Next twice.

Figure 12.4. Select Novell Business Continuity Cluster as well as the iManager Snapins for NetWare for installation.

[View full size image]

3.
Now you need to enter the distinguished name of the first cluster object you want to add to the BCC environment (see Figure 12.5). Select cluster1 and then click Next to continue.

Figure 12.5. Add the clusters you want to put in the BCC environment one by one.

[View full size image]

4.
Make sure that all nodes in the cluster are selected, and click Next to continue.

5.
Next, you need to select the NetWare server where you want to install the iManager snapins. Select bcc1-node2 and click Next to continue. This will install all required software to the nodes in the first cluster.

6.
After installation, the line LDBCC.NCF is added to the AUTOEXEC.NCF file on all nodes in the cluster. Open the AUTOEXEC.NCF file and comment out this line with the REM statement or the pound sign (#).

7.
Repeat the preceding steps for the other cluster.

Configure the Identity Manager eDirectory Drivers

Now that all required software components have been installed, it is time to start the real work. The first part of this is to configure the Identity Manager drivers. You need to configure Identity Manager on both sides of the cluster. On site1 the driver is responsible for keeping a copy of all relevant objects from site2, and on site2 the Identity Manager driver must make sure that all relevant objects from site1 are kept. For this purpose, you need to configure separate containers that function as a landing zone for the objects from the other site at both sites. In this example, we will work with ou=fromsite2.ou=site1.o=bcc and ou=fromsite1.ou=site2.o=bcc.

Note

The relevant snapins for management of BCC are not installed on all servers in your tree by default. From now on, we recommend that you connect only to the servers where the snapins are installed.

1.
Before proceeding, make sure that the servers on which you want to configure iManager have an eDirectory replica of the partition they are in. Then start iManager at https://yourserver/nps.

2.
Browse to DirXML Utilities and select New Driver. Then select In a New Driver Set and click Next to continue.

3.
Enter the following information for the new Driver set and click Next to continue (see Figure 12.6):

Figure 12.6. Enter the name, context, and responsible server to create an Identity Manager driver.

[View full size image]

  • Name: BCC site1-site2

  • Context: site1.bcc

  • Server: bcc1-node2.site1.bcc

  • Uncheck the Create a New Partition on This Driver Set box

4.
In the next screen you need to create the driver. Creating an Identity Manager driver all by yourself is a daunting task, so select Import a Driver Configuration from the Server (.XML File) as shown in Figure 12.7, and from the drop-down list select BCCClusterResourceSynchronization.xml. Then click Next to continue.

Figure 12.7. Import the BCCClusterResourceSynchronization driver.

[View full size image]

5.
Now you have to enter specific properties for the driver. Use the following information:

  • Driver name: site12site2 driver.

  • Name of the SSL certificate: none.

Note

If you are synchronizing between two different eDirectory trees, make sure that you do select an SSL certificate here. In this scenario, however, we are synchronizing within the same eDirectory tree, and therefore an SSL certificate is not needed.

  • Name of other DirXML node: 192.168.2.120 (this should be the IP address of the node in cluster 2 where you have installed the Identity Manager software).

  • Port number for this driver: 2002.

  • Full distinguished name of this cluster: cluster1.site1.bcc.

  • Full distinguished name of the other cluster: cluster2.site2.bcc.

  • Context where cluster-enabled pool and volume objects will be synchronized for this cluster: fromsite2.site1.bcc.

  • Parent container context of cluster-enabled pool and volume objects for other cluster: site2.bcc. This is the context in the other tree where the cluster-enabled pools and volumes reside. Normally, this is the same context as the context where your cluster object is created.

  • Name of the other cluster's tree: BCC-TREE.

6.
Now select Define Security Equivalences and browse to the admin object in your tree. This ensures that the Identity Manager driver has sufficient rights to create objects in your tree when synchronizing between two clusters. Then click Next to continue.

7.
You will now see a summary of the current driver configuration (see Figure 12.8). The summary gives an overview of all eDirectory objects that are created. From this overview, click Finish to add the driver to your network.

Figure 12.8. In the summary screen an overview is given of the driver as it will be created in eDirectory.

[View full size image]

Tip

For novice users, the Identity Manager summary view does not provide a great level of detail. To get more insight in how the driver is configured, click Finish with Overview.

8.
Repeat all tasks to create an Identity Manager driver in site2 as well. Make sure that in the driver details screen, the following information is used:

  • Driver name: site22site1 driver.

  • Name of the SSL certificate: none.

  • Name of other DirXML node: 192.168.2.20 (this should be the IP address of the node in cluster 1 where you have installed the Identity Manager software).

  • Port number for this driver: 2002.

  • Full distinguished name of this cluster: cluster2.site2.bcc.

  • Full distinguished name of the other cluster: cluster1.site1.bcc.

  • Context where cluster-enabled pool and volume objects will be synchronized for this cluster: fromsite1.site2.bcc.

  • Parent container context of cluster-enabled pool and volume objects for other cluster: site1.bcc. This is the context in the other tree where the cluster-enabled pools and volumes reside. Normally, this is the same context as the context where your cluster object is created.

  • Name of the other cluster's tree: BCC-TREE.

Start the Identity Manager Drivers, NCS, and BCC

Now that the Identity Manager drivers have been created, they still need to be started. The tasks described next need to be performed on both servers where Identity Manager is installed:

1.
From iManager, select DirXML, DirXML Overview. Then select Search Entire Tree and click the Search button. This will display a screen as shown in Figure 12.9.

Figure 12.9. The DirXML Overview screen provides an overview of all Identity Manager drivers currently present in your eDirectory tree.

[View full size image]

2.
Click the stop sign on the driver and select Start Driver. The driver will now be started. If it has started successfully, you will see a yin/yang-icon on the driver to indicate that it is running. After the first driver is up, do the same for the second driver.

Tip

If you are not really into Identity Manager, troubleshooting it is difficult. If you can't get the drivers synchronizing to each other and you are sure that you have completed all steps as listed in this procedure, we recommend that you delete the drivers and create new drivers. This will be a lot easier than troubleshooting where exactly you have made a mistake in configuring the IDM drivers.

3.
Only after the Identity Manager drivers have started successfully, start Novell Cluster Services with the LDNCS.NCF file on all nodes in both clusters. If you have created the INITSTRT.NCF file as described previously, don't forget to run it before you start the cluster.

4.
When NCS has loaded, start BCC by running the LDBCC.NCF file on all nodes in both clusters. This should get your BCC environment up and running.

Configuring BCC

Now that all the required software is active, it is time to make some modifications to all related eDirectory objects. First, you will learn how to enable Business Continuity Features for both clusters:

1.
Start iManager on the node in cluster 1 that is configured with Identity Manager. Then select Clusters, Cluster Options and browse to cluster 1.

2.
Click Properties, and then select the Business Continuity tab.

3.
Click the Enable Business Continuity Features check box as shown in Figure 12.10 and click OK.

Figure 12.10. When all necessary software components are loaded, you can enable Business Continuity features for both clusters.

[View full size image]

4.
Repeat this procedure for the other cluster in your setup.

Now that BCC has been enabled for both clusters, you need to set up authentication between the two clusters. To make this possible, you must add the administrator username and password that your cluster will use to connect to the other cluster. Of course, this needs to happen on both clusters.

Tip

If you have problems setting up the connection between both clusters, always make sure that the Identity Manager driver is active first. Without a working Identity Manager driver, you can't get a working BCC connection between two clusters. Use the DirXML, DirXML Overview option in iManager to monitor current activity of Identity Manager drivers.

Here are the steps to take:

1.
From the Business Continuity properties tab of the selected cluster, in the Connections section select the other cluster and click Edit (see Figure 12.11).

Figure 12.11. To set up a connection between two clusters, you need to enter authentication credentials.

[View full size image]

2.
Enter the complete username including the eDirectory context of your admin user and its password.

3.
Repeat this step for the other cluster.

Tip

If the other cluster does not appear in the list of possible peer clusters, make sure that the following conditions have been met:

  • The BCC software must be installed.

  • Identity Manager must be installed.

  • The other cluster must be resolvable to the other clusters and eDirectory trees.

  • The BCC DirXML drivers must be configured and running.

  • The other cluster must be enabled for BCC.

Tip

Still having problems setting up the connection? You can use the cluster credentials command from the BCC console on the servers to set up the connection as well. On bcc1-node 2 enter the command cluster credentials cluster2. Next, enter the distinguished admin name and password. Repeat this procedure on the other node by using the command cluster credentials cluster1.

Next, some objects need to be created in the clusters and they need to be configured for failover. The following procedure shows how to create a volume pool resource and how this object is enabled for failover in a BCC environment:

1. | On bcc1-node2, load NSSMU. Start NSSMU, select pools, and press Insert to create a new pool. Give the pool the name site1pool and press Enter.
2. | Select the appropriate partition on the iSCSI shared disk device and press Enter. Next, accept the default for the partition size and press Enter.
3. | Accept all default values for the pool object and provide it a unique IP address. Make sure that the IP address can be resolved to the name of this pool in the /etc/hosts file on your server. Then click Create to create the pool.
4. | Now create a volume named site1vol in the pool you have just created.
5. | Repeat this procedure to also create a pool and volume object for site2 in your cluster.

Next, you have to make the partitions available across the clusters in your network. For this purpose, you need to mirror the partitions. The next procedure shows how to do this with NSSMU:

1.
Load NSSMU on bcc1-node2 and select Partitions.

2.
Select the NSS partition with the site1pool pool on it.

3.
Press F3 to display mirror options. It will show a list of all available partitions as shown in Figure 12.12. Now select the right partition object. You need to look at the partition ID to determine which partition you need. In the Device Name value, each partition has a name like [V060-A3-D3:0]. Determine the device name of the partition on the local server that you want to mirror, and next determine the device name of the partition on the other server you have just created. That leaves two devices available for mirroring. If you created a 400MB partition on the local server and, directly after that, the 410MB partition on the other server to serve as a partition for mirroring, you have to look at the local device name plus 1; if the local device name is [V060-A3-D3:0], the device name you need on the other server is [V060-A3-D4:0]. Select this partition, press Enter, and when asked Create Mirror on Shared Device? select Yes to continue. In a few seconds the mirror will be established. You can see this on the partition object as well because it will have a mirror ID added to it.

Figure 12.12. After successfully creating the mirror pair, the mirrored partition will have a mirror ID added to it.

[View full size image]

4.
Use the MIRROR STATUS command on the console of your server to monitor the mirroring process. Wait for it to complete and then continuethis can take a few minuteswith the next phase of the procedure.

With the creation of the pool objects, for both pools a virtual server object is created in your clusters automatically. In the next phase of this procedure, we need to set up the virtual server object for use in your BCC environment:

1.
In iManager, select Cluster, Cluster Options. Then select the SITE1POOL_SERVER object and click Properties.

2.
Activate the Business Continuity tab, and there check the Enable Business Continuity Features check box.

3.
Now from the Resource Script Replacements section, click New.

4.
For the search value enter 192.168.2.51 (the address of the local cluster-enabled pool object), and for the replace value enter 192.168.2.152. The second IP address is the IP address that will be added to the pool object you have just created after it fails over successfully.

5.
Now from the list of Available Peer Clusters, select both cluster1 and cluster2 and click OK (see Figure 12.13). You have just BCC-enabled your cluster pool object.

Figure 12.13. To enable a resource for BCC migration, you need to select the cluster peers for this resource, and you have to add search and replace values that can be applied to the cluster load scripts.

[View full size image]

6.
Repeat this procedure for the SITE2POOL_SERVER object. On this object, use 192.168.2.151 and 192.168.2.52 as the search and replace values.

You have just set up your cluster-enabled pool objects for BCC. Now it is time to test whether it works. For this purpose, you are going to select the CLUSTER2POOL_SERVER object and migrate it to the other cluster.

1. | From iManager, select BCC Manager and select cluster2.
2. | Check the SITE2POOL_SERVER object and click BCC Migrate to migrate. This allows you to migrate the resource to any cluster that is available from the list of available peers (see Figure 12.14).

Figure 12.14. From BCC Manager, select the resource you want to migrate to another cluster and then click BCC Migrate.

[View full size image]


Summary

If you need more than a cluster, Novell Business Continuity Cluster Services is a great solution; it allows you to create a cluster of clusters. In this chapter you read about the components BCC is made of, and you created a simple BCC lab setup. Although it is simple, it involves a lot of steps to be performed. Nevertheless, after it works, BCC offers a great solution for optimized fault tolerance within large enterprise environments. Watch out for BCC version 1.1 because it will also allow you to synchronize between OES Linux clusters.


Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [Y] [Z]


A

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [Y] [Z]

access control, iSCSI software on Linux
access rights, assigning for iSCSI initiators
across-the-wire migrations
Active-Active clusters
Active-Passive clusters
add secondary ipaddress command
alert command
Alert state
alerting [See configuring, email notification.]
Apache daemon, configuring
applications
cluster design and
cluster-enabling [See cluster-enabling applications, GroupWise.]
clustering [See clusters, backups.]
installing
NCF or Linux script files
on shared disk
architecture of Backup Express
arrays
authentication keys
availability
defined
high availability
increasing
with environment
with hardware
with procedures
with software
average downtime, calculating


B

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [Y] [Z]

backup clients
Backup Express
backups
Novell backup software
SMS (Storage Management Services)
Syncsort Backup Express
BCC (Business Continuity Clustering)
advantages of
configuring
installing
operational overview
test environment, building
BCC configuration
BCC installation
Identity Manager driver configuration
Identity Manager installation
NetWare and Novell Cluster Services installation
requirements
starting Identity Manager drivers
usage scenarios
bus (SCSI)
Business Continuity Clustering [See BCC (Business Continuity Clustering).]


C

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [Y] [Z]

caching mode (GroupWise)
CDBs (Command Descriptor Blocks)
Censustaker protocol
certificates, configuring for iFolder
cl status command
CLSTRLIB (cluster configuration library)
cluster command
cluster containers
defined
parameters
cluster design
application installation
NCF or Linux script files
on shared disk
application selection
disk configurations
hardware redundant disk configurations
single-disk configurations
software redundant disk configurations
eDirectory
failover matrix
file system
heartbeat LANs
NIC teaming
number of nodes
Split Brain Detector partition, mirroring
storage connection methods
Fibre Channel
SCSI
without shared storage
Cluster Event Log (iManager)
Cluster Management Port parameter
Cluster Manager screen (iManager)
cluster master resource
cluster membership views
cluster nodes
IP addresses
local files, eDirectory objects and
cluster of clusters [See BCC (Business Continuity Clustering).]
Cluster Options screen (iManager)
Cluster Resource Manager (CRM)
Cluster Services Management Agent (CMA)
Cluster System Services (CSS)
cluster templates 2nd
cluster virtual server object (eDirectory)
cluster virtual servers
Cluster Volume Broker
cluster volumes 2nd 3rd
NSS pools, creating
NSS volumes, creating
shared disk partitions, creating
cluster-enabling applications
GroupWise
on Linux
on NetWare
iFolder
Samba
Squid proxy server
when to use
clusters [See also Novell Cluster Services, for BCC test environment.]
backups
Novell backup software
SMS (Storage Management Services)
Syncsort Backup Express
defined
documentation, creating
eDirectory objects
cluster containers
cluster nodes
cluster resources
cluster templates
cluster volume resources
local cluster node files and
high availability between
installing
on Linux
on NetWare
iSCSI setup 2nd
managing
with iManager
with Remote Manager
selecting in iManager
states of
terminology
troubleshooting
types of
upgrading
across-the-wire migrations
in-place upgrades
installing from scratch
migrating to new cluster
NetWare 5.1 to NetWare 6.5
NetWare 6 to NetWare 6.5
NetWare 6.5 to Linux
when not to use
CMA (Cluster Services Management Agent)
Comatose state
Command Descriptor Blocks (CDBs)
command line
iSCSI initiator connection startup 2nd
managing Novell Cluster Services
starting and stopping Novell Cluster Services
on Linux
on NetWare
commands, iSCSI targets on NetWare
configuring
Apache daemon
BCC for BCC test environment
certificates for iFolder
DNS servers
email notification
Heartbeat software (Linux)
Identity Manager drivers for BCC test environment
iFolder
iSCSI initiator software on Linux
load and unload scripts
for GroupWise 2nd
for iFolder
NFS servers for Heartbeat clusters
shared storage for Heartbeat software
smb.conf file
connection startup for iSCSI initiators
connections, iSCSI targets
ConsoleOne management application
consolidation (iSCSI)
convert command
CRM (Cluster Resource Manager)
CSS (Cluster System Services)
cvb rebuild command
cvsbind command


D

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [Y] [Z]

database version (GroupWise), setting
del secondary ipaddress command
design [See cluster design, application installation, NCF or Linux script files.]
device servers in Backup Express
dhcp command
DHCP servers, clustering
on Linux
on NetWare
directories, sharing with Samba
DirXML [See Identity Manager.]
disk configurations (cluster design)
hardware redundant disk configurations
single-disk configurations
software redundant disk configurations
Distributed Replicated Block Device (DRBD) 2nd
DNS servers
load balancing
when to cluster-enable
documentation, creating
down command
downed clusters, upgrading with
downtime
average downtime, calculating
avoiding [See availability, defined.]
maximum downtime for percentage of availability
DRBD (Distributed Replicated Block Device) 2nd
drivers (Identity Manager)
configuring for BCC test environment
starting for BCC test environment


E

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [Y] [Z]

eDirectory
cluster design
objects
cluster containers
cluster nodes
cluster resources
cluster templates
cluster volume resources
local cluster node files and
support for
editing
haresources file
sample configuration files
email notification, configuring
environment, increasing availability with
epoch
epoch number
Event log, troubleshooting cluster resources
EVMS volume manager
EVMS volumes
creating 2nd
installing
exit on error command
ext3 file system


F

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [Y] [Z]

failback
failback mode in cluster resource objects
failover
failover clusters
failover matrix, creating
failover mode in cluster resource objects
Fibre Channel
file system (cluster design)
File System Factory
file systems
ext3
for Linux cluster volumes
NSS 2nd
Reiser


G

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [Y] [Z]

GIPC (Group Interprocess Protocol)
Group Anchor protocol
Group Interprocess Protocol (GIPC)
GroupWise, cluster-enabling
on Linux
on NetWare


H

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [Y] [Z]

ha.cf file, tuning
hardware redundant disk configurations (cluster design)
hardware requirements for Novell Cluster Services
hardware, increasing availability with
haresources file, editing
heartbeat LANs
Heartbeat package (SUSE Linux Enterprise Server)
heartbeat packets
Heartbeat protocol
Heartbeat Settings parameter
Heartbeat software (Linux)
configuring
configuring shared storage
installing
managing
Novell Cluster Services versus
split brain, avoiding
heartbeats
defined
IP addresses for, changing
high availability
between clusters
defined
home directories [See updating, user home directories.]
hot spare disks
hot-pluggable disks


I

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [Y] [Z]

Identity Manager
configuring drivers for BCC test environment
installing for BCC test environment
starting drivers for BCC test environment
Identity Manager for BCC test environment
iFolder on cluster nodes
iFolder, cluster-enabling
ignore error command
iManager 2nd
Cluster Event Log
Cluster Manager screen
Cluster Options screen
plug-ins, installing
selecting cluster resources
SMS configuration
iManager plug-ins
in-place upgrades
NetWare 5.1 to NetWare 6.5
NetWare 6 to NetWare 6.5
info command
initiators (iSCSI)
on Linux
access control setup
configuring software
connecting to targets
installing iSCSI software
monitoring
on NetWare
assigning access rights
connection startup
monitoring
renaming NetWare servers
requirements
installing
applications
NCF or Linux script files
on shared disk
Backup Express
on Linux
on NetWare
BCC for BCC test environment
cluster hardware/software from scratch
cluster resource templates
cluster resources
on Linux
on NetWare
EVMS volumes
GroupWise
on Linux
on NetWare
IP addresses
for heartbeats, changing
GroupWise clustering and
master IP addresses
of cluster master resource, changing
secondary IP addresses
IQNs (iSCSI Qualified Names)
iSCSI
advantages of
as OS-independent storage
as remote storage
for offsite storage mirroring
initiators
on Linux
on NetWare
renaming NetWare servers
requirements
naming conventions
network design guidelines
operational overview
RFCs for
sessions, monitoring
shared storage setup 2nd
support for
targets
LDAP security
on NetWare
requirements
ISCSI commands
iSCSI Qualified Names (IQNs)
iSCSI software on Linux
iSCSI target software
ISCSINIT commands
ISCSITAR commands


J

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [Y] [Z]

join command


L

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [Y] [Z]

LANs, heartbeat LANs
LDAP (Lightweight Directory Access Protocol) 2nd
leave command
licenses, adding to nodes
Lightweight Directory Access Protocol (LDAP) 2nd
Linux
Backup Express, installing
creating clustered volumes
NSS pools, creating
NSS volumes, creating
shared disk partitions, creating
file systems for cluster volumes
GroupWise, cluster-enabling
Heartbeat software
configuring
configuring shared storage
installing
managing
Novell Cluster Services versus
split brain, avoiding
iFolder, cluster-enabling
installing cluster resources
installing Novell Cluster Services
iSCSI initiators on
access control setup
configuring software
connecting to targets
installing iSCSI software
monitoring
migrating from NetWare to
migrating from Windows to
nbackup utility
nodes, removing
Samba, cluster-enabling
SBD partition, creating new
script files, executing from load and unload scripts
SMS configuration files
Squid proxy server, cluster-enabling
starting and stopping Novell Cluster Services
SUSE Linux Enterprise Server, Heartbeat package
troubleshooting cluster resources
upgrading NetWare 6.5 to
lives
load balancing
load scripts
configuring
for GroupWise 2nd
for iFolder
defined
executing NCF or Linux script files
in cluster resource objects
NCS-related commands
local cluster node files, eDirectory objects and
local storage access (iSCSI)
Logger Screen (NetWare)
logical disks
LVM volume manager


M

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [Y] [Z]

maintenance command
management applications
managing
Backup Express
cluster resources
with iManager
with Remote Manager
Heartbeat software (Linux)
iSCSI sessions
Novell Cluster Services
email notification
from command line
manually starting and stopping Novell Cluster Services
master IP addresses
master nodes
defined
removing
master servers in Backup Express
maximum downtime for percentage of availability
Mean Time Between Failure (MTBF)
Mean Time to Repair (MTTR)
Membership protocol
migrate command
migrating [See upgrading clusters, across-the-wire migrations.]
migration
Migration Wizard
mirroring
offsite storage mirroring, iSCSI for
Split Brain Detector partition
mixed-cluster environments, upgrading NetWare 6.5 to Linux
modules in Novell Cluster Services
CLSTRLIB
Cluster Volume Broker
CMA
CRM
CSS
GIPC
PCLUSTER
SBD
VLL
monitoring [See managing, Backup Express.]
mount points, creating
MSAP (Multiple Server Access Prevention)
MTBF (Mean Time Between Failure)
MTTR (Mean Time to Repair)
Multiple Server Access Prevention (MSAP)
multiple-site environment (BCC)


N

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [Y] [Z]

naming conventions (iSCSI)
nbackup utility
NCF files, executing from load and unload scripts
NCS [See Novell Cluster Services, for BCC test environment.]
NDS Sync state
NetWare
Backup Express, installing
creating clustered volumes
NSS pools, creating
NSS volumes, creating
shared disk partitions, creating
GroupWise, cluster-enabling
installing cluster resources
installing for BCC test environment
installing Novell Cluster Services
iSCSI initiators on
assigning access rights
connection startup
monitoring
iSCSI targets on
commands for
creating target partition
installing software
monitoring
Logger Screen
managing cluster resources with Remote Manager
migrating from Windows to
migrating to Linux
migrating to NetWare
SBCON.NLM backup software
SBD partition, creating new
servers, renaming
SMS configuration files
starting and stopping Novell Cluster Services
NetWare 5.1, upgrading to NetWare 6.5
NetWare 6, upgrading to NetWare 6.5
NetWare 6.5
upgrading NetWare 5.1 to
upgrading NetWare 6 to
upgrading to Linux
NetWare for BCC test environment
NetWare nodes
adding
to Linux clusters
to NetWare clusters
removing 2nd
network design, iSCSI guidelines
NFS servers, configuring for Heartbeat clusters
NIC teaming
nodes
cluster node objects (eDirectory)
defined
licenses
Linux nodes, removing
master nodes, removing
NetWare nodes
adding to Linux clusters
adding to NetWare clusters
removing 2nd
number of
slave nodes, removing
notification (email), configuring
Novell Cluster Services [See also clusters, backups.]
benefits of
for BCC test environment
Heartbeat software (Linux) versus
in VMware
installing
for BCC test environment
in VMware
on Linux
on NetWare
managing
email notification
from command line
modules in
CLSTRLIB
Cluster Volume Broker
CMA
CRM
CSS
GIPC
PCLUSTER
SBD
VLL
on Linux
on NetWare
operational overview
requirements
secondary IP addresses
shared disk access
starting and stopping
manually
on Linux
on NetWare
terminology
Novell Identity Manager
Novell Server Consolidation and Migration Toolkit
NSS
cluster-enabled volume objects (eDirectory)
file system 2nd
pools, creating
volumes, creating


O

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [Y] [Z]

objects [See eDirectory, objects, cluster containers.]
OES NetWare [See NetWare, Backup Express, installing.]
offline command
Offline state
offsite storage mirroring, iSCSI for
online command
OS-independent storage, iSCSI as


P

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [Y] [Z]

Panning protocol
parameters of set command
partitions
iSCSI target partitions, creating
SBD partition, creating new
shared disk partitions [See shared disks, access, ; partitions, shared disk partitions.]
PCLUSTER (Portal Cluster Agent)
PDUs (Protocol Data Units)
Perl CLI (command-line interface) utilities
plug-ins for iManager, installing
POA (Post Office Agent), GroupWise clustering
poison pills 2nd
PolyServe Matrix Server
pools command
port numbers, GroupWise clustering and
Portal Cluster Agent (PCLUSTER)
Post Office Agent (POA), GroupWise clustering
post offices
adding
to GroupWise on Linux 2nd
to GroupWise on NetWare
Post Office Agent (POA), GroupWise clustering
setting database version
procedures, increasing availability with
Protocol Data Units (PDUs)
proxy servers, Squid


Q

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [Y] [Z]

quorum
Quorum Trigger parameter
Quorum wait state


R

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [Y] [Z]

RAID (redundant array of independent disks)
RAID 0
RAID 0+1
RAID 1
RAID 10
RAID 5
rcheartbeat command
redundant array of independent disks (RAID)
redundant disk configurations (cluster design)
hardware
software
references (server), updating
Reiser file system
Remote Manager
iSCSI initiator connection startup
monitoring iSCSI sessions
remote storage, iSCSI as
removing
Linux nodes
master nodes
NetWare nodes 2nd
slave nodes
renaming NetWare servers
replicas, eDirectory cluster design
requirements
BCC test environment
iSCSI initiators
iSCSI targets
Novell Cluster Services
resources command
restart command
restarting SMDR daemon
RFCs for iSCSI
rights, assigning for iSCSI initiators
rolling upgrades 2nd
round robin configuration
rsync synchronization
Running state


S

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [Y] [Z]

Samba, cluster-enabling on Linux
sample configuration files, editing
SAN (storage area network) 2nd
SBCON.NLM backup software
SBD (Split Brain Detector) 2nd
creating new
defined
mirroring
size of
scan for new devices command
SCSI (Small Computer System Interface) 2nd
secondary IP addresses
security
authentication keys
LDAP security, iSCSI targets and
selecting cluster resources in iManager
Sequencer protocol
Server Consolidation Utility
as part of Novell Server Consolidation and Migration
Server Health Utility
server references, updating
servers
backup clients
device servers
master servers
NetWare servers, renaming
sessions (iSCSI), monitoring
set command
shared disks
access 2nd
installing applications on
partitions, creating
VMware [See VMware, ESX Server clustering configurations.]
shared resources, creating
shared storage
cluster design without
configuring for Heartbeat software (Linux)
creating
disk configurations
hardware redundant disk
single-disk configurations
software redundant disk
file system
GroupWise clustering and
iSCSI as
iSCSI setup
storage connection methods
Fibre Channel
SCSI
sharing directories with Samba
single-disk configurations (cluster design)
slave nodes
defined
removing
Small Computer Systems Interface (SCSI) 2nd
smb.conf file, configuring
SMDR (Storage Management Data Requester)
restarting SMDR daemon
SMS (Storage Management Services)
software
increasing availability with
mirroring
Novell Cluster Services software requirements
software redundant disk configurations (cluster design)
Split Brain Detector [See modules in Novell Cluster Services, SBD.]
split brain situations
split brain, avoiding in Heartbeat clusters
Squid proxy server, cluster-enabling on Linux
starting
connections for iSCSI initiators
Identity Manager drivers for BCC test environment
Novell Cluster Services
manually
on Linux
on NetWare
stats command
status command
Stonith devices
stopping Novell Cluster Services
manually
on Linux
on NetWare
storage area network (SAN) 2nd
storage connection methods (cluster design)
Fibre Channel
SCSI
Storage Management Data Requester (SMDR)
restarting SMDR daemon
Storage Management Services (SMS)
storage, iSCSI [See iSCSI, advantages of.]
stretched clusters
SUSE Linux [See Linux, Backup Express, installing.]
SUSE Linux Enterprise Server, Heartbeat package
synchronization
Identity Manager [See Identity Manager.]
rsync
transitive synchronization
Syncsort Backup Express
system requirements [See requirements, BCC test environment.]


T

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [Y] [Z]

Target Service Agents (TSAs)
targets (iSCSI)
connecting to
LDAP security
on NetWare
commands for
creating target partition
installing software
monitoring
requirements
templates [See cluster templates.]
terminology in Novell Cluster Services
test environment (BCC) 2nd
BCC configuration
BCC installation
Identity Manager driver configuration
Identity Manager installation
NetWare and Novell Cluster Services installation
requirements
starting Identity Manager drivers
testing
GroupWise
on Linux
on NetWare
iFolder on Linux
timeout
tolerance rate
transitive synchronization
troubleshooting cluster resources
TSAs (Target Service Agents)
tuning ha.cf file
two-site environment (BCC)


U

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [Y] [Z]

Unassigned state
unload scripts
configuring
for GroupWise 2nd
for iFolder
defined
executing NCF or Linux script files
in cluster resource objects
NCS-related commands
Unloading state
updating
server references
user home directories
upgrading clusters
across-the-wire migrations
in-place upgrades
installing from scratch
migrating to new cluster
NetWare 5.1 to NetWare 6.5
NetWare 6 to NetWare 6.5
NetWare 6.5 to Linux
uptime [See availability, defined.]
user home directories, updating


V

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [Y] [Z]

view command
virtual environment [See VMware, ESX Server clustering configurations.]
virtual hosts, configuring iFolder server as
Virtual Interface Architecture Link Layer (VLL)
virtual nodes
virtual servers [See cluster virtual servers.]
VLL (Virtual Interface Architecture Link Layer)
VMware
ESX Server clustering configurations
GSX Server clustering configurations
installing Novell Cluster Services
Workstation clustering configurations
volume managers, EVMS versus LVM
volumes
cluster volumes [See cluster volumes.]
EVMS volumes
creating 2nd
installing
NSS cluster-enabled volume objects (eDirectory)


W

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [Y] [Z]

WebAccess, GroupWise clustering
Windows, migrating to NetWare or Linux


Y

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [Y] [Z]

Yes,Tested and Approved program (Novell)


Z

Previous Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [Y] [Z]

ZENworks for Desktops, when to cluster-enable